== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 17:35:51 (1746135351) start mds service on oleg324-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing load_modules_local oleg324-server: Loading modules from /home/green/git/lustre-release/lustre oleg324-server: detected 4 online CPUs by sysfs oleg324-server: MODOPTS_LIBCFS= oleg324-server: Force libcfs to create 2 CPU partitions oleg324-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg324-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg324-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg324-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg324-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg324-server: oleg324-server.virtnet: executing set_default_debug -1 all pdsh@oleg324-client: oleg324-server: ssh exited with exit code 1 Started lustre-OST0000 oleg324-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg324-client.virtnet: -o user_xattr,flock 192.168.203.124@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 186 (time 1746135407.32 total 10.76 last 17.29) - open/close 414 (time 1746135418.18 total 21.62 last 20.99) - open/close 652 (time 1746135429.10 total 32.54 last 21.80) - open/close 855 (time 1746135439.82 total 43.26 last 18.94) - open/close 1162 (time 1746135449.85 total 53.29 last 30.62) - open/close 1369 (time 1746135460.87 total 64.31 last 18.78) - open/close 1608 (time 1746135471.16 total 74.61 last 23.22) - open/close 1831 (time 1746135481.52 total 84.96 last 21.53) - open/close 2042 (time 1746135492.43 total 95.87 last 19.35) - open/close 2270 (time 1746135502.43 total 105.87 last 22.78) - open/close 2414 (time 1746135513.03 total 116.48 last 13.58) - open/close 2644 (time 1746135523.26 total 126.70 last 22.50) - open/close 2832 (time 1746135534.07 total 137.51 last 17.40) - open/close 3138 (time 1746135544.07 total 147.51 last 30.58) - open/close 3392 (time 1746135554.50 total 157.94 last 24.37) - open/close 3690 (time 1746135565.45 total 168.90 last 27.19) - open/close 4005 (time 1746135575.76 total 179.20 last 30.58) - open/close 4238 (time 1746135585.78 total 189.22 last 23.25) - open/close 4540 (time 1746135596.60 total 200.04 last 27.91) - open/close 4731 (time 1746135606.62 total 210.06 last 19.06) - open/close 4961 (time 1746135617.59 total 221.03 last 20.96) - open/close 5221 (time 1746135628.05 total 231.49 last 24.86) - open/close 5515 (time 1746135638.72 total 242.16 last 27.56) - open/close 5727 (time 1746135649.60 total 253.04 last 19.48) - open/close 5901 (time 1746135659.74 total 263.18 last 17.16) - open/close 6167 (time 1746135670.47 total 273.91 last 24.79) - open/close 6475 (time 1746135681.24 total 284.68 last 28.59) - open/close 6677 (time 1746135692.07 total 295.51 last 18.65) - open/close 6973 (time 1746135703.05 total 306.49 last 26.96) - open/close 7207 (time 1746135713.05 total 316.49 last 23.40) - open/close 7376 (time 1746135723.38 total 326.82 last 16.36) - open/close 7556 (time 1746135733.39 total 336.83 last 17.99) - open/close 7803 (time 1746135744.21 total 347.66 last 22.81) - open/close 8021 (time 1746135754.58 total 358.02 last 21.02) - open/close 8186 (time 1746135767.53 total 370.97 last 12.74) - open/close 8436 (time 1746135778.30 total 381.74 last 23.22) - open/close 8657 (time 1746135788.47 total 391.91 last 21.74) - open/close 8916 (time 1746135799.44 total 402.88 last 23.61) - open/close 9211 (time 1746135809.47 total 412.91 last 29.41) - open/close 9406 (time 1746135819.48 total 422.92 last 19.48) - open/close 9670 (time 1746135830.19 total 433.64 last 24.63) - open/close 9902 (time 1746135840.68 total 444.13 last 22.12) - open/close 10000 (time 1746135844.86 total 448.30 last 23.45) - open/close 11242 (time 1746135854.87 total 458.31 last 124.10) - open/close 12691 (time 1746135864.88 total 468.32 last 144.84) - open/close 14124 (time 1746135874.88 total 478.33 last 143.17) - open/close 15377 (time 1746135884.89 total 488.33 last 125.22) - open/close 16513 (time 1746135894.90 total 498.35 last 113.45) - open/close 17735 (time 1746135904.92 total 508.36 last 122.07) - open/close 18929 (time 1746135914.92 total 518.36 last 119.36) - open/close 20000 (time 1746135923.50 total 526.94 last 124.76) - open/close 21475 (time 1746135933.51 total 536.95 last 147.42) - open/close 23001 (time 1746135943.51 total 546.95 last 152.59) - open/close 24248 (time 1746135953.51 total 556.95 last 124.68) - open/close 25527 (time 1746135963.51 total 566.95 last 127.85) - open/close 26857 (time 1746135973.52 total 576.96 last 132.89) - open/close 28398 (time 1746135983.53 total 586.97 last 154.05) - open/close 30000 (time 1746135992.53 total 595.97 last 177.90) - open/close 31872 (time 1746136002.54 total 605.98 last 187.12) - open/close 33314 (time 1746136012.54 total 615.98 last 144.09) - open/close 34685 (time 1746136022.55 total 625.99 last 137.00) - open/close 36067 (time 1746136032.56 total 636.00 last 138.12) - open/close 37263 (time 1746136042.56 total 646.00 last 119.56) - open/close 38781 (time 1746136052.57 total 656.01 last 151.69) - open/close 40000 (time 1746136062.20 total 665.64 last 126.57) - open/close 41278 (time 1746136072.21 total 675.65 last 127.69) - open/close 42485 (time 1746136082.21 total 685.65 last 120.61) - open/close 43641 (time 1746136092.22 total 695.66 last 115.52) - open/close 44750 (time 1746136102.22 total 705.67 last 110.84) - open/close 45861 (time 1746136112.23 total 715.67 last 111.07) - open/close 47034 (time 1746136122.23 total 725.67 last 117.28) - open/close 48134 (time 1746136132.24 total 735.68 last 109.88) - open/close 49298 (time 1746136142.24 total 745.68 last 116.39) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 748.86 seconds: 66.28 ops/second - unlinked 0 (time 1746136147 ; total 0 ; last 0) - unlinked 10000 (time 1746136211 ; total 64 ; last 64) - unlinked 20000 (time 1746136251 ; total 104 ; last 40) - unlinked 30000 (time 1746136294 ; total 147 ; last 43) - unlinked 40000 (time 1746136330 ; total 183 ; last 36) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 224 seconds: 221.571426 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg324-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg324-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg324-server stop mds service on oleg324-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg324-server unloading modules via unload_modules_local on: 'oleg324-server' oleg324-server: oleg324-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-client: ssh exited with exit code 2 pdsh@oleg324-client: oleg324-server: ssh exited with exit code 2