== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 02:45:26 (1761288326) start mds service on oleg152-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing load_modules_local oleg152-server: Loading modules from /home/green/git/lustre-release/lustre oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1034: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1035: echo: write error: Device or resource busy oleg152-server: detected 4 online CPUs by sysfs oleg152-server: MODOPTS_LIBCFS= oleg152-server: Force libcfs to create 2 CPU partitions oleg152-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg152-server: quota/lquota options: 'hash_lqs_cur_bits=3' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg152-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg152-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg152-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg152-server: oleg152-server.virtnet: executing set_default_debug -1 all pdsh@oleg152-client: oleg152-server: ssh exited with exit code 1 Started lustre-OST0000 oleg152-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg152-client.virtnet: -o user_xattr,flock 192.168.201.152@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 144 (time 1761288394.62 total 10.70 last 13.46) - open/close 264 (time 1761288404.68 total 20.75 last 11.94) - open/close 366 (time 1761288415.29 total 31.36 last 9.61) - open/close 490 (time 1761288425.31 total 41.38 last 12.38) - open/close 585 (time 1761288436.29 total 52.36 last 8.65) - open/close 769 (time 1761288446.30 total 62.37 last 18.38) - open/close 1006 (time 1761288456.92 total 72.99 last 22.31) - open/close 1221 (time 1761288466.99 total 83.06 last 21.35) - open/close 1454 (time 1761288477.09 total 93.16 last 23.07) - open/close 1662 (time 1761288487.74 total 103.81 last 19.53) - open/close 1841 (time 1761288498.11 total 114.18 last 17.26) - open/close 2013 (time 1761288508.52 total 124.59 last 16.52) - open/close 2210 (time 1761288519.15 total 135.22 last 18.54) - open/close 2451 (time 1761288529.15 total 145.22 last 24.09) - open/close 2657 (time 1761288539.89 total 155.96 last 19.18) - open/close 2879 (time 1761288549.90 total 165.97 last 22.18) - open/close 3050 (time 1761288559.90 total 175.98 last 17.09) - open/close 3264 (time 1761288570.73 total 186.80 last 19.77) - open/close 3508 (time 1761288581.81 total 197.88 last 22.02) - open/close 3739 (time 1761288591.86 total 207.93 last 23.00) - open/close 3961 (time 1761288602.79 total 218.87 last 20.30) - open/close 4190 (time 1761288612.80 total 228.87 last 22.88) - open/close 4392 (time 1761288622.80 total 238.87 last 20.20) - open/close 4594 (time 1761288632.81 total 248.88 last 20.19) - open/close 4749 (time 1761288643.12 total 259.20 last 15.03) - open/close 4960 (time 1761288653.74 total 269.81 last 19.88) - open/close 5200 (time 1761288664.18 total 280.25 last 22.97) - open/close 5410 (time 1761288674.66 total 290.73 last 20.04) - open/close 5620 (time 1761288685.30 total 301.37 last 19.75) - open/close 5791 (time 1761288695.31 total 311.38 last 17.09) - open/close 6026 (time 1761288706.15 total 322.22 last 21.67) - open/close 6189 (time 1761288716.83 total 332.90 last 15.26) - open/close 6328 (time 1761288726.89 total 342.96 last 13.82) - open/close 6387 (time 1761288736.89 total 352.96 last 5.90) - open/close 6575 (time 1761288747.82 total 363.89 last 17.21) - open/close 6728 (time 1761288757.87 total 373.94 last 15.22) - open/close 6941 (time 1761288767.87 total 383.94 last 21.29) - open/close 7247 (time 1761288778.59 total 394.66 last 28.55) - open/close 7494 (time 1761288788.60 total 404.67 last 24.68) - open/close 7805 (time 1761288798.61 total 414.68 last 31.07) - open/close 8041 (time 1761288808.67 total 424.74 last 23.45) - open/close 8337 (time 1761288819.11 total 435.18 last 28.36) - open/close 8624 (time 1761288829.86 total 445.93 last 26.70) - open/close 8878 (time 1761288840.37 total 456.44 last 24.17) - open/close 9146 (time 1761288850.93 total 467.00 last 25.37) - open/close 9437 (time 1761288861.54 total 477.61 last 27.43) - open/close 9709 (time 1761288872.17 total 488.24 last 25.58) - open/close 9948 (time 1761288883.75 total 499.82 last 20.63) - open/close 10254 (time 1761288894.63 total 510.70 last 28.14) - open/close 10554 (time 1761288905.26 total 521.33 last 28.22) - open/close 10856 (time 1761288916.01 total 532.08 last 28.09) - open/close 11127 (time 1761288926.31 total 542.38 last 26.31) - open/close 11330 (time 1761288936.99 total 553.07 last 19.00) - open/close 11612 (time 1761288947.56 total 563.63 last 26.69) - open/close 11909 (time 1761288958.12 total 574.19 last 28.12) - open/close 12203 (time 1761288968.42 total 584.49 last 28.56) - open/close 12545 (time 1761288978.47 total 594.54 last 34.02) - open/close 12767 (time 1761288988.64 total 604.71 last 21.82) - open/close 13383 (time 1761288998.65 total 614.73 last 61.52) - open/close 14329 (time 1761289008.66 total 624.73 last 94.56) - open/close 15329 (time 1761289018.66 total 634.73 last 99.99) - open/close 16269 (time 1761289028.67 total 644.75 last 93.87) - open/close 17202 (time 1761289038.68 total 654.75 last 93.22) - open/close 18270 (time 1761289048.69 total 664.76 last 106.77) - open/close 19578 (time 1761289058.69 total 674.76 last 130.79) - open/close 20000 (time 1761289062.18 total 678.25 last 120.65) - open/close 20977 (time 1761289072.19 total 688.26 last 97.66) - open/close 21856 (time 1761289082.19 total 698.27 last 87.84) - open/close 22734 (time 1761289092.20 total 708.27 last 87.78) - open/close 23724 (time 1761289102.21 total 718.28 last 98.90) - open/close 24778 (time 1761289112.21 total 728.29 last 105.32) - open/close 25783 (time 1761289122.22 total 738.29 last 100.41) - open/close 27055 (time 1761289132.23 total 748.30 last 127.15) - open/close 28447 (time 1761289142.23 total 758.30 last 139.17) - open/close 29615 (time 1761289152.23 total 768.30 last 116.80) - open/close 30000 (time 1761289155.73 total 771.80 last 110.05) - open/close 31107 (time 1761289165.74 total 781.81 last 110.62) - open/close 32305 (time 1761289175.74 total 791.81 last 119.77) - open/close 33598 (time 1761289185.74 total 801.81 last 129.26) - open/close 35060 (time 1761289195.74 total 811.81 last 146.15) - open/close 36381 (time 1761289205.75 total 821.82 last 132.06) - open/close 38019 (time 1761289215.75 total 831.82 last 163.75) - open/close 39348 (time 1761289225.76 total 841.83 last 132.82) - open/close 40000 (time 1761289231.41 total 847.49 last 115.24) - open/close 41164 (time 1761289241.41 total 857.49 last 116.39) - open/close 42361 (time 1761289251.42 total 867.49 last 119.66) - open/close 43853 (time 1761289261.42 total 877.49 last 149.13) - open/close 45210 (time 1761289271.43 total 887.50 last 135.66) - open/close 46704 (time 1761289281.43 total 897.50 last 149.37) - open/close 48260 (time 1761289291.43 total 907.50 last 155.58) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No space left on device total: 49632 open/close in 916.51 seconds: 54.15 ops/second - unlinked 0 (time 1761289302 ; total 0 ; last 0) - unlinked 10000 (time 1761289364 ; total 62 ; last 62) - unlinked 20000 (time 1761289400 ; total 98 ; last 36) - unlinked 30000 (time 1761289435 ; total 133 ; last 35) - unlinked 40000 (time 1761289470 ; total 168 ; last 35) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49632) error: No such file or directory total: 49632 unlinks in 199 seconds: 249.407028 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg152-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg152-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg152-server stop mds service on oleg152-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg152-server /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy unloading modules via unload_modules_local on: 'oleg152-server' oleg152-server: oleg152-server.virtnet: executing unload_modules_local oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 114: echo: write error: Device or resource busy oleg152-server: /home/green/git/lustre-release/lustre/scripts/lustre_rmmod: line 156: echo: write error: Device or resource busy modules unloaded. pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-client: ssh exited with exit code 2 pdsh@oleg152-client: oleg152-server: ssh exited with exit code 2