== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:41:54 (1746139314) start mds service on oleg321-server Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing load_modules_local oleg321-server: Loading modules from /home/green/git/lustre-release/lustre oleg321-server: detected 4 online CPUs by sysfs oleg321-server: MODOPTS_LIBCFS= oleg321-server: Force libcfs to create 2 CPU partitions oleg321-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg321-server: quota/lquota options: 'hash_lqs_cur_bits=3' Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg321-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg321-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg321-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg321-server: oleg321-server.virtnet: executing set_default_debug -1 all pdsh@oleg321-client: oleg321-server: ssh exited with exit code 1 Started lustre-OST0000 oleg321-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg321-client.virtnet: -o user_xattr,flock 192.168.203.121@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 370 (time 1746139370.84 total 10.40 last 35.57) - open/close 597 (time 1746139380.95 total 20.51 last 22.45) - open/close 842 (time 1746139390.96 total 30.52 last 24.48) - open/close 1040 (time 1746139401.27 total 40.84 last 19.20) - open/close 1149 (time 1746139412.17 total 51.73 last 10.00) - open/close 1459 (time 1746139422.18 total 61.75 last 30.96) - open/close 1677 (time 1746139432.18 total 71.75 last 21.80) - open/close 1922 (time 1746139442.20 total 81.76 last 24.46) - open/close 2105 (time 1746139452.71 total 92.27 last 17.41) - open/close 2309 (time 1746139463.36 total 102.93 last 19.14) - open/close 2502 (time 1746139474.36 total 113.92 last 17.56) - open/close 2767 (time 1746139485.70 total 125.27 last 23.36) - open/close 3041 (time 1746139496.02 total 135.58 last 26.55) - open/close 3277 (time 1746139506.08 total 145.64 last 23.46) - open/close 3395 (time 1746139516.99 total 156.55 last 10.82) - open/close 3521 (time 1746139527.95 total 167.52 last 11.49) - open/close 3764 (time 1746139538.62 total 178.19 last 22.78) - open/close 3904 (time 1746139550.50 total 190.07 last 11.78) - open/close 4063 (time 1746139560.52 total 200.08 last 15.88) - open/close 4352 (time 1746139571.22 total 210.79 last 27.00) - open/close 4602 (time 1746139581.89 total 221.45 last 23.44) - open/close 4746 (time 1746139592.06 total 231.63 last 14.15) - open/close 4954 (time 1746139602.70 total 242.26 last 19.56) - open/close 5187 (time 1746139612.79 total 252.35 last 23.10) - open/close 5448 (time 1746139623.03 total 262.60 last 25.47) - open/close 5678 (time 1746139633.34 total 272.91 last 22.31) - open/close 5908 (time 1746139643.35 total 282.92 last 22.98) - open/close 6020 (time 1746139655.58 total 295.15 last 9.16) - open/close 6186 (time 1746139666.21 total 305.78 last 15.62) - open/close 6439 (time 1746139676.66 total 316.23 last 24.21) - open/close 6627 (time 1746139690.66 total 330.22 last 13.43) - open/close 6841 (time 1746139700.69 total 340.26 last 21.32) - open/close 7009 (time 1746139712.58 total 352.15 last 14.13) - open/close 7283 (time 1746139722.68 total 362.24 last 27.14) - open/close 7461 (time 1746139732.69 total 372.26 last 17.77) - open/close 7658 (time 1746139743.32 total 382.89 last 18.53) - open/close 7846 (time 1746139754.21 total 393.78 last 17.27) - open/close 8083 (time 1746139764.56 total 404.12 last 22.91) - open/close 8823 (time 1746139774.56 total 414.12 last 73.98) - open/close 10000 (time 1746139784.55 total 424.11 last 117.83) - open/close 11202 (time 1746139794.55 total 434.12 last 120.17) - open/close 12516 (time 1746139804.55 total 444.12 last 131.35) - open/close 13839 (time 1746139814.56 total 454.12 last 132.26) - open/close 15154 (time 1746139824.56 total 464.12 last 131.47) - open/close 16435 (time 1746139834.56 total 474.13 last 128.08) - open/close 18047 (time 1746139844.56 total 484.13 last 161.18) - open/close 19368 (time 1746139854.57 total 494.13 last 132.05) - open/close 20000 (time 1746139859.10 total 498.67 last 139.23) - open/close 21662 (time 1746139869.11 total 508.67 last 166.17) - open/close 23546 (time 1746139879.11 total 518.67 last 188.39) - open/close 25253 (time 1746139889.11 total 528.68 last 170.63) - open/close 26463 (time 1746139899.12 total 538.68 last 120.94) - open/close 27899 (time 1746139909.12 total 548.68 last 143.59) - open/close 29341 (time 1746139919.12 total 558.68 last 144.17) - open/close 30000 (time 1746139922.87 total 562.43 last 175.83) - open/close 31439 (time 1746139932.87 total 572.44 last 143.81) - open/close 32752 (time 1746139942.87 total 582.44 last 131.30) - open/close 33877 (time 1746139952.88 total 592.45 last 112.36) - open/close 35263 (time 1746139962.89 total 602.45 last 138.53) - open/close 36449 (time 1746139972.89 total 612.46 last 118.57) - open/close 37720 (time 1746139982.90 total 622.47 last 126.97) - open/close 39027 (time 1746139992.90 total 632.47 last 130.68) - open/close 40000 (time 1746139999.88 total 639.45 last 139.37) - open/close 41320 (time 1746140009.89 total 649.45 last 131.94) - open/close 42603 (time 1746140019.90 total 659.46 last 128.21) - open/close 43781 (time 1746140029.90 total 669.46 last 117.79) - open/close 45215 (time 1746140039.90 total 679.47 last 143.35) - open/close 46375 (time 1746140049.90 total 689.47 last 115.95) - open/close 47465 (time 1746140059.91 total 699.47 last 108.99) - open/close 48611 (time 1746140069.91 total 709.48 last 114.53) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 716.17 seconds: 69.30 ops/second - unlinked 0 (time 1746140078 ; total 0 ; last 0) - unlinked 10000 (time 1746140146 ; total 68 ; last 68) - unlinked 20000 (time 1746140198 ; total 120 ; last 52) - unlinked 30000 (time 1746140247 ; total 169 ; last 49) - unlinked 40000 (time 1746140298 ; total 220 ; last 51) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 268 seconds: 185.194031 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg321-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg321-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg321-server stop mds service on oleg321-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg321-server unloading modules via unload_modules_local on: 'oleg321-server' oleg321-server: oleg321-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-client: ssh exited with exit code 2 pdsh@oleg321-client: oleg321-server: ssh exited with exit code 2