== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:43 (1746157603) start mds service on oleg229-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing load_modules_local oleg229-server: Loading modules from /home/green/git/lustre-release/lustre oleg229-server: detected 4 online CPUs by sysfs oleg229-server: MODOPTS_LIBCFS= oleg229-server: Force libcfs to create 2 CPU partitions oleg229-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg229-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg229-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg229-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg229-server: oleg229-server.virtnet: executing set_default_debug -1 all pdsh@oleg229-client: oleg229-server: ssh exited with exit code 1 Started lustre-OST0000 oleg229-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg229-client.virtnet: -o user_xattr,flock 192.168.202.129@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 356 (time 1746157640.59 total 10.69 last 33.31) - open/close 559 (time 1746157650.60 total 20.69 last 20.30) - open/close 734 (time 1746157660.83 total 30.93 last 17.09) - open/close 980 (time 1746157672.10 total 42.20 last 21.83) - open/close 1331 (time 1746157683.04 total 53.14 last 32.08) - open/close 1586 (time 1746157693.52 total 63.61 last 24.35) - open/close 1827 (time 1746157703.96 total 74.05 last 23.08) - open/close 2074 (time 1746157713.97 total 84.06 last 24.69) - open/close 2337 (time 1746157724.63 total 94.72 last 24.66) - open/close 2602 (time 1746157735.53 total 105.62 last 24.31) - open/close 2810 (time 1746157746.04 total 116.13 last 19.79) - open/close 3063 (time 1746157756.73 total 126.82 last 23.67) - open/close 3322 (time 1746157767.02 total 137.11 last 25.16) - open/close 3568 (time 1746157777.49 total 147.58 last 23.49) - open/close 3764 (time 1746157788.28 total 158.37 last 18.17) - open/close 3995 (time 1746157798.28 total 168.37 last 23.09) - open/close 4184 (time 1746157808.79 total 178.88 last 17.98) - open/close 4434 (time 1746157818.80 total 188.89 last 24.99) - open/close 4568 (time 1746157828.80 total 198.89 last 13.40) - open/close 4704 (time 1746157839.72 total 209.81 last 12.45) - open/close 4923 (time 1746157850.66 total 220.75 last 20.02) - open/close 5155 (time 1746157861.63 total 231.72 last 21.14) - open/close 5296 (time 1746157871.78 total 241.88 last 13.88) - open/close 5434 (time 1746157881.96 total 252.05 last 13.56) - open/close 5697 (time 1746157892.43 total 262.52 last 25.12) - open/close 5967 (time 1746157902.65 total 272.74 last 26.42) - open/close 6179 (time 1746157913.41 total 283.51 last 19.69) - open/close 6381 (time 1746157923.42 total 293.51 last 20.19) - open/close 6542 (time 1746157935.21 total 305.30 last 13.66) - open/close 6741 (time 1746157945.25 total 315.34 last 19.81) - open/close 7042 (time 1746157955.26 total 325.35 last 30.08) - open/close 7295 (time 1746157965.78 total 335.87 last 24.05) - open/close 7516 (time 1746157975.78 total 345.87 last 22.09) - open/close 7701 (time 1746157985.79 total 355.88 last 18.49) - open/close 7920 (time 1746157996.34 total 366.44 last 20.74) - open/close 8065 (time 1746158006.36 total 376.45 last 14.48) - open/close 8304 (time 1746158016.64 total 386.73 last 23.24) - open/close 8371 (time 1746158026.71 total 396.80 last 6.65) - open/close 8596 (time 1746158037.59 total 407.69 last 20.67) - open/close 8758 (time 1746158049.42 total 419.51 last 13.70) - open/close 9532 (time 1746158059.43 total 429.52 last 77.37) - open/close 10000 (time 1746158063.97 total 434.06 last 102.99) - open/close 11078 (time 1746158073.97 total 444.06 last 107.78) - open/close 12272 (time 1746158083.98 total 454.07 last 119.33) - open/close 13410 (time 1746158093.98 total 464.07 last 113.74) - open/close 14667 (time 1746158103.99 total 474.08 last 125.63) - open/close 15864 (time 1746158113.99 total 484.08 last 119.66) - open/close 17190 (time 1746158124.00 total 494.09 last 132.53) - open/close 18396 (time 1746158134.00 total 504.09 last 120.57) - open/close 19480 (time 1746158144.00 total 514.09 last 108.37) - open/close 20000 (time 1746158147.82 total 517.91 last 136.11) - open/close 21173 (time 1746158157.83 total 527.92 last 117.23) - open/close 22117 (time 1746158167.84 total 537.93 last 94.31) - open/close 23027 (time 1746158177.84 total 547.93 last 91.00) - open/close 24056 (time 1746158187.85 total 557.94 last 102.82) - open/close 25088 (time 1746158197.86 total 567.95 last 103.04) - open/close 26103 (time 1746158207.87 total 577.96 last 101.44) - open/close 27152 (time 1746158217.88 total 587.97 last 104.80) - open/close 28277 (time 1746158227.89 total 597.98 last 112.35) - open/close 29559 (time 1746158237.89 total 607.99 last 128.15) - open/close 30000 (time 1746158241.67 total 611.76 last 116.86) - open/close 31217 (time 1746158251.67 total 621.76 last 121.70) - open/close 32688 (time 1746158261.67 total 631.76 last 147.03) - open/close 33748 (time 1746158271.67 total 641.77 last 105.97) - open/close 34989 (time 1746158281.68 total 651.77 last 124.05) - open/close 36456 (time 1746158291.68 total 661.77 last 146.65) - open/close 37811 (time 1746158301.68 total 671.78 last 135.48) - open/close 39297 (time 1746158311.69 total 681.78 last 148.55) - open/close 40000 (time 1746158317.16 total 687.25 last 128.48) - open/close 41357 (time 1746158327.16 total 697.25 last 135.67) - open/close 42642 (time 1746158337.16 total 707.25 last 128.49) - open/close 43961 (time 1746158347.17 total 717.26 last 131.86) - open/close 45366 (time 1746158357.17 total 727.26 last 140.49) - open/close 47187 (time 1746158367.18 total 737.27 last 181.89) - open/close 48409 (time 1746158377.18 total 747.27 last 122.15) - open/close 49801 (time 1746158387.18 total 757.28 last 139.16) total: 50000 open/close in 758.64 seconds: 65.91 ops/second - unlinked 0 (time 1746158389 ; total 0 ; last 0) - unlinked 10000 (time 1746158449 ; total 60 ; last 60) - unlinked 20000 (time 1746158499 ; total 110 ; last 50) - unlinked 30000 (time 1746158561 ; total 172 ; last 62) - unlinked 40000 (time 1746158628 ; total 239 ; last 67) total: 50000 unlinks in 306 seconds: 163.398697 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg229-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg229-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg229-server stop mds service on oleg229-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg229-server unloading modules via unload_modules_local on: 'oleg229-server' oleg229-server: oleg229-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-client: ssh exited with exit code 2 pdsh@oleg229-client: oleg229-server: ssh exited with exit code 2