== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:23:26 (1746138206) start mds service on oleg248-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions libkmod: kmod_module_get_holders: could not open '/sys/module/acpi_cpufreq/holders': No such file or directory ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing load_modules_local oleg248-server: Loading modules from /home/green/git/lustre-release/lustre oleg248-server: detected 4 online CPUs by sysfs oleg248-server: MODOPTS_LIBCFS= oleg248-server: Force libcfs to create 2 CPU partitions oleg248-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg248-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg248-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg248-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg248-server: oleg248-server.virtnet: executing set_default_debug -1 all pdsh@oleg248-client: oleg248-server: ssh exited with exit code 1 Started lustre-OST0000 oleg248-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg248-client.virtnet: -o user_xattr,flock 192.168.202.148@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 160 (time 1746138249.60 total 10.01 last 15.99) - open/close 463 (time 1746138260.34 total 20.75 last 28.21) - open/close 686 (time 1746138270.82 total 31.23 last 21.28) - open/close 789 (time 1746138281.10 total 41.51 last 10.02) - open/close 913 (time 1746138291.81 total 52.21 last 11.58) - open/close 1138 (time 1746138302.78 total 63.19 last 20.50) - open/close 1375 (time 1746138312.78 total 73.19 last 23.70) - open/close 1650 (time 1746138323.19 total 83.60 last 26.41) - open/close 1979 (time 1746138334.13 total 94.54 last 30.08) - open/close 2235 (time 1746138344.45 total 104.86 last 24.80) - open/close 2460 (time 1746138354.46 total 114.87 last 22.48) - open/close 2727 (time 1746138365.07 total 125.47 last 25.17) - open/close 3002 (time 1746138376.08 total 136.49 last 24.97) - open/close 3215 (time 1746138387.04 total 147.45 last 19.43) - open/close 3459 (time 1746138397.95 total 158.36 last 22.37) - open/close 3629 (time 1746138408.61 total 169.02 last 15.95) - open/close 3885 (time 1746138418.62 total 179.03 last 25.57) - open/close 4147 (time 1746138429.32 total 189.73 last 24.48) - open/close 4289 (time 1746138439.69 total 200.10 last 13.70) - open/close 4528 (time 1746138449.69 total 210.10 last 23.89) - open/close 4736 (time 1746138459.70 total 220.11 last 20.78) - open/close 5006 (time 1746138470.32 total 230.73 last 25.43) - open/close 5251 (time 1746138480.32 total 240.73 last 24.49) - open/close 5403 (time 1746138490.74 total 251.14 last 14.60) - open/close 5641 (time 1746138500.74 total 261.14 last 23.80) - open/close 5933 (time 1746138511.29 total 271.70 last 27.67) - open/close 6167 (time 1746138522.21 total 282.62 last 21.42) - open/close 6419 (time 1746138532.76 total 293.17 last 23.90) - open/close 6625 (time 1746138542.76 total 303.17 last 20.60) - open/close 6796 (time 1746138553.25 total 313.66 last 16.30) - open/close 6985 (time 1746138563.79 total 324.20 last 17.93) - open/close 7219 (time 1746138574.68 total 335.09 last 21.49) - open/close 7463 (time 1746138585.31 total 345.72 last 22.96) - open/close 7744 (time 1746138595.83 total 356.24 last 26.72) - open/close 8024 (time 1746138606.70 total 367.11 last 25.76) - open/close 8277 (time 1746138617.37 total 377.78 last 23.71) - open/close 8511 (time 1746138627.58 total 387.99 last 22.92) - open/close 8801 (time 1746138638.20 total 398.61 last 27.30) - open/close 9049 (time 1746138648.75 total 409.16 last 23.51) - open/close 9275 (time 1746138658.82 total 419.23 last 22.44) - open/close 9615 (time 1746138668.83 total 429.24 last 33.98) - open/close 10000 (time 1746138678.08 total 438.48 last 41.64) - open/close 11730 (time 1746138688.08 total 448.49 last 172.98) - open/close 13574 (time 1746138698.08 total 458.49 last 184.37) - open/close 15869 (time 1746138708.08 total 468.49 last 229.44) - open/close 17624 (time 1746138718.08 total 478.49 last 175.49) - open/close 19166 (time 1746138728.08 total 488.49 last 154.16) - open/close 20000 (time 1746138733.80 total 494.21 last 145.89) - open/close 21461 (time 1746138743.81 total 504.22 last 146.01) - open/close 22811 (time 1746138753.82 total 514.22 last 134.88) - open/close 24176 (time 1746138763.83 total 524.24 last 136.31) - open/close 25550 (time 1746138773.83 total 534.24 last 137.39) - open/close 27114 (time 1746138783.83 total 544.24 last 156.35) - open/close 28857 (time 1746138793.84 total 554.25 last 174.19) - open/close 30000 (time 1746138801.70 total 562.10 last 145.47) - open/close 31544 (time 1746138811.70 total 572.11 last 154.36) - open/close 32948 (time 1746138821.71 total 582.12 last 140.29) - open/close 34195 (time 1746138831.71 total 592.12 last 124.63) - open/close 35514 (time 1746138841.71 total 602.12 last 131.89) - open/close 36973 (time 1746138851.71 total 612.12 last 145.89) - open/close 38499 (time 1746138861.72 total 622.13 last 152.56) - open/close 39811 (time 1746138871.72 total 632.13 last 131.17) - open/close 41119 (time 1746138881.73 total 642.13 last 130.73) - open/close 42657 (time 1746138891.73 total 652.13 last 153.79) - open/close 44311 (time 1746138901.73 total 662.14 last 165.32) - open/close 46242 (time 1746138911.74 total 672.15 last 192.94) - open/close 47855 (time 1746138921.74 total 682.15 last 161.27) - open/close 49450 (time 1746138931.74 total 692.15 last 159.48) total: 50000 open/close in 695.94 seconds: 71.85 ops/second - unlinked 0 (time 1746138937 ; total 0 ; last 0) - unlinked 10000 (time 1746138987 ; total 50 ; last 50) - unlinked 20000 (time 1746139027 ; total 90 ; last 40) - unlinked 30000 (time 1746139061 ; total 124 ; last 34) - unlinked 40000 (time 1746139088 ; total 151 ; last 27) total: 50000 unlinks in 188 seconds: 265.957458 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg248-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg248-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg248-server stop mds service on oleg248-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg248-server unloading modules via unload_modules_local on: 'oleg248-server' oleg248-server: oleg248-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-client: ssh exited with exit code 2 pdsh@oleg248-client: oleg248-server: ssh exited with exit code 2