== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 07:10:07 (1739535007) start mds service on oleg253-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing load_modules_local oleg253-server: Loading modules from /home/green/git/lustre-release/lustre oleg253-server: detected 4 online CPUs by sysfs oleg253-server: MODOPTS_LIBCFS= oleg253-server: Force libcfs to create 2 CPU partitions oleg253-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg253-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg253-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 254 (time 1739535046.69 total 10.00 last 25.39) - open/close 504 (time 1739535057.70 total 21.01 last 22.72) - open/close 685 (time 1739535068.06 total 31.37 last 17.46) - open/close 955 (time 1739535078.25 total 41.56 last 26.51) - open/close 1239 (time 1739535088.66 total 51.97 last 27.27) - open/close 1477 (time 1739535098.68 total 61.98 last 23.77) - open/close 1725 (time 1739535109.57 total 72.88 last 22.76) - open/close 2011 (time 1739535119.58 total 82.88 last 28.59) - open/close 2280 (time 1739535130.15 total 93.46 last 25.44) - open/close 2436 (time 1739535140.15 total 103.46 last 15.60) - open/close 2738 (time 1739535150.75 total 114.06 last 28.47) - open/close 3045 (time 1739535161.24 total 124.55 last 29.27) - open/close 3354 (time 1739535171.75 total 135.06 last 29.41) - open/close 3764 (time 1739535182.34 total 145.65 last 38.72) - open/close 4102 (time 1739535193.13 total 156.44 last 31.32) - open/close 4375 (time 1739535204.01 total 167.32 last 25.09) - open/close 4560 (time 1739535214.41 total 177.72 last 17.80) - open/close 4810 (time 1739535224.41 total 187.72 last 24.99) - open/close 4962 (time 1739535234.51 total 197.82 last 15.05) - open/close 5219 (time 1739535245.41 total 208.72 last 23.56) - open/close 5357 (time 1739535255.42 total 218.73 last 13.80) - open/close 5496 (time 1739535265.52 total 228.82 last 13.76) - open/close 5698 (time 1739535275.89 total 239.20 last 19.47) - open/close 5878 (time 1739535286.17 total 249.48 last 17.51) - open/close 6093 (time 1739535298.80 total 262.11 last 17.02) - open/close 6309 (time 1739535310.45 total 273.76 last 18.53) - open/close 6590 (time 1739535320.72 total 284.03 last 27.37) - open/close 6867 (time 1739535330.87 total 294.18 last 27.29) - open/close 7111 (time 1739535340.87 total 304.18 last 24.39) - open/close 7417 (time 1739535351.28 total 314.59 last 29.40) - open/close 7693 (time 1739535361.56 total 324.87 last 26.85) - open/close 7921 (time 1739535371.61 total 334.91 last 22.70) - open/close 8159 (time 1739535381.61 total 344.92 last 23.79) - open/close 8460 (time 1739535392.13 total 355.44 last 28.62) - open/close 8712 (time 1739535402.14 total 365.45 last 25.18) - open/close 8971 (time 1739535412.15 total 375.45 last 25.88) - open/close 9373 (time 1739535422.15 total 385.46 last 40.18) - open/close 10000 (time 1739535426.42 total 389.72 last 146.98) - open/close 11511 (time 1739535436.42 total 399.73 last 151.08) - open/close 12964 (time 1739535446.42 total 409.73 last 145.23) - open/close 14471 (time 1739535456.43 total 419.73 last 150.65) - open/close 15915 (time 1739535466.43 total 429.73 last 144.38) - open/close 17544 (time 1739535476.43 total 439.74 last 162.89) - open/close 19295 (time 1739535486.43 total 449.74 last 174.99) - open/close 20000 (time 1739535490.21 total 453.52 last 186.70) - open/close 21610 (time 1739535500.21 total 463.52 last 160.93) - open/close 23132 (time 1739535510.21 total 473.52 last 152.20) - open/close 24540 (time 1739535520.21 total 483.52 last 140.77) - open/close 25892 (time 1739535530.22 total 493.52 last 135.20) - open/close 27303 (time 1739535540.22 total 503.53 last 141.02) - open/close 29011 (time 1739535550.22 total 513.53 last 170.75) - open/close 30000 (time 1739535554.61 total 517.92 last 225.60) - open/close 31824 (time 1739535564.61 total 527.92 last 182.36) - open/close 33541 (time 1739535574.61 total 537.92 last 171.62) - open/close 35483 (time 1739535584.62 total 547.93 last 194.11) - open/close 37389 (time 1739535594.62 total 557.93 last 190.57) - open/close 39360 (time 1739535604.63 total 567.94 last 196.97) - open/close 40000 (time 1739535607.60 total 570.90 last 215.59) - open/close 42122 (time 1739535617.60 total 580.91 last 212.13) - open/close 44062 (time 1739535627.60 total 590.91 last 193.93) - open/close 45952 (time 1739535637.60 total 600.91 last 188.98) - open/close 48063 (time 1739535647.61 total 610.92 last 211.03) total: 50000 open/close in 619.98 seconds: 80.65 ops/second - unlinked 0 (time 1739535657 ; total 0 ; last 0) - unlinked 10000 (time 1739535693 ; total 36 ; last 36) - unlinked 20000 (time 1739535724 ; total 67 ; last 31) - unlinked 30000 (time 1739535752 ; total 95 ; last 28) - unlinked 40000 (time 1739535778 ; total 121 ; last 26) total: 50000 unlinks in 147 seconds: 340.136047 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2