== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 10:35:01 (1786804501) start mds service on oleg411-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing load_modules_local oleg411-server: Loading modules from /home/green/git/lustre-release/lustre oleg411-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg411-server: detected 4 online CPUs by sysfs oleg411-server: MODOPTS_LIBCFS= oleg411-server: Force libcfs to create 2 CPU partitions oleg411-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg411-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg411-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg411-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg411-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg411-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg411-server: oleg411-server.virtnet: executing set_default_debug -1 all pdsh@oleg411-client: oleg411-server: ssh exited with exit code 1 Started lustre-OST0000 oleg411-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg411-client.virtnet: -o user_xattr,flock 192.168.204.111@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 273 (time 1786804626.49 total 10.00 last 27.29) - open/close 542 (time 1786804637.17 total 20.68 last 25.19) - open/close 800 (time 1786804647.33 total 30.84 last 25.40) - open/close 1055 (time 1786804658.41 total 41.92 last 23.01) - open/close 1290 (time 1786804668.76 total 52.27 last 22.71) - open/close 1475 (time 1786804678.87 total 62.38 last 18.29) - open/close 1557 (time 1786804688.90 total 72.42 last 8.17) - open/close 1731 (time 1786804699.48 total 83.00 last 16.45) - open/close 1935 (time 1786804709.49 total 93.00 last 20.39) - open/close 2093 (time 1786804719.49 total 103.00 last 15.80) - open/close 2350 (time 1786804730.08 total 113.60 last 24.25) - open/close 2547 (time 1786804740.10 total 123.61 last 19.67) - open/close 2756 (time 1786804750.10 total 133.62 last 20.90) - open/close 2987 (time 1786804760.11 total 143.63 last 23.07) - open/close 3243 (time 1786804771.13 total 154.64 last 23.25) - open/close 3492 (time 1786804783.65 total 167.16 last 19.89) - open/close 3743 (time 1786804793.65 total 177.16 last 25.09) - open/close 3882 (time 1786804803.66 total 187.18 last 13.88) - open/close 4125 (time 1786804814.31 total 197.82 last 22.83) - open/close 4183 (time 1786804824.60 total 208.12 last 5.63) - open/close 4396 (time 1786804834.62 total 218.13 last 21.27) - open/close 4594 (time 1786804845.67 total 229.18 last 17.92) - open/close 4733 (time 1786804855.91 total 239.42 last 13.57) - open/close 4965 (time 1786804866.05 total 249.56 last 22.88) - open/close 5227 (time 1786804876.05 total 259.57 last 26.19) - open/close 5467 (time 1786804886.32 total 269.83 last 23.38) - open/close 5752 (time 1786804896.34 total 279.85 last 28.45) - open/close 6008 (time 1786804906.71 total 290.23 last 24.67) - open/close 6216 (time 1786804917.67 total 301.18 last 18.98) - open/close 6416 (time 1786804928.54 total 312.05 last 18.40) - open/close 6619 (time 1786804938.68 total 322.19 last 20.03) - open/close 6854 (time 1786804948.68 total 332.20 last 23.48) - open/close 7110 (time 1786804958.71 total 342.22 last 25.54) - open/close 7241 (time 1786804969.73 total 353.25 last 11.88) - open/close 7375 (time 1786804980.77 total 364.28 last 12.15) - open/close 7515 (time 1786804991.15 total 374.66 last 13.49) - open/close 7770 (time 1786805001.15 total 384.66 last 25.49) - open/close 7968 (time 1786805011.38 total 394.90 last 19.34) - open/close 8183 (time 1786805021.99 total 405.50 last 20.27) - open/close 8314 (time 1786805032.01 total 415.53 last 13.07) - open/close 8422 (time 1786805042.74 total 426.26 last 10.06) - open/close 8507 (time 1786805052.89 total 436.40 last 8.38) - open/close 8753 (time 1786805062.90 total 446.41 last 24.57) - open/close 8970 (time 1786805073.07 total 456.59 last 21.33) - open/close 9124 (time 1786805083.50 total 467.01 last 14.78) - open/close 9259 (time 1786805094.44 total 477.95 last 12.34) - open/close 9468 (time 1786805104.45 total 487.96 last 20.87) - open/close 9691 (time 1786805115.10 total 498.61 last 20.94) - open/close 9968 (time 1786805125.54 total 509.06 last 26.53) - open/close 10201 (time 1786805135.54 total 519.06 last 23.30) - open/close 10408 (time 1786805145.70 total 529.21 last 20.38) - open/close 10530 (time 1786805155.70 total 539.22 last 12.20) - open/close 10717 (time 1786805166.30 total 549.81 last 17.64) - open/close 10983 (time 1786805176.31 total 559.82 last 26.59) - open/close 11152 (time 1786805186.95 total 570.46 last 15.88) - open/close 11437 (time 1786805197.85 total 581.36 last 26.15) - open/close 11639 (time 1786805208.87 total 592.38 last 18.33) - open/close 11763 (time 1786805218.88 total 602.39 last 12.39) - open/close 11924 (time 1786805228.90 total 612.41 last 16.08) - open/close 12011 (time 1786805239.60 total 623.11 last 8.13) - open/close 12256 (time 1786805249.63 total 633.15 last 24.41) - open/close 12307 (time 1786805260.45 total 643.96 last 4.72) - open/close 12447 (time 1786805270.46 total 653.97 last 13.99) - open/close 12573 (time 1786805280.46 total 663.97 last 12.60) - open/close 12735 (time 1786805290.89 total 674.40 last 15.54) - open/close 12846 (time 1786805301.54 total 685.05 last 10.42) - open/close 13034 (time 1786805311.98 total 695.49 last 18.01) - open/close 13232 (time 1786805322.21 total 705.72 last 19.35) - open/close 13480 (time 1786805332.70 total 716.22 last 23.63) - open/close 13669 (time 1786805342.71 total 726.23 last 18.88) - open/close 13900 (time 1786805352.72 total 736.23 last 23.09) - open/close 14130 (time 1786805362.73 total 746.24 last 22.97) - open/close 14335 (time 1786805373.61 total 757.13 last 18.83) - open/close 14589 (time 1786805383.69 total 767.20 last 25.22) - open/close 14815 (time 1786805393.69 total 777.21 last 22.58) - open/close 15050 (time 1786805404.09 total 787.61 last 22.60) - open/close 15191 (time 1786805414.12 total 797.63 last 14.06) - open/close 15418 (time 1786805424.29 total 807.80 last 22.32) - open/close 15653 (time 1786805434.53 total 818.05 last 22.94) - open/close 15925 (time 1786805444.54 total 828.05 last 27.19) - open/close 16145 (time 1786805454.71 total 838.22 last 21.64) - open/close 16403 (time 1786805467.50 total 851.01 last 20.17) - open/close 16655 (time 1786805477.50 total 861.01 last 25.20) - open/close 16876 (time 1786805487.71 total 871.22 last 21.66) - open/close 17082 (time 1786805500.10 total 883.62 last 16.62) - open/close 17321 (time 1786805510.45 total 893.96 last 23.10) - open/close 17582 (time 1786805520.99 total 904.50 last 24.77) - open/close 17792 (time 1786805531.50 total 915.01 last 19.99) - open/close 18075 (time 1786805542.54 total 926.05 last 25.62) - open/close 18295 (time 1786805552.72 total 936.23 last 21.61) - open/close 18514 (time 1786805563.32 total 946.84 last 20.65) - open/close 18645 (time 1786805573.74 total 957.25 last 12.57) - open/close 18794 (time 1786805585.27 total 968.79 last 12.92) - open/close 18976 (time 1786805596.24 total 979.75 last 16.60) - open/close 19164 (time 1786805608.90 total 992.41 last 14.85) - open/close 19237 (time 1786805619.18 total 1002.69 last 7.10) - open/close 19848 (time 1786805629.19 total 1012.70 last 61.06) - open/close 20000 (time 1786805631.66 total 1015.17 last 61.49) - open/close 20768 (time 1786805641.67 total 1025.18 last 76.74) - open/close 21592 (time 1786805651.68 total 1035.19 last 82.32) - open/close 22302 (time 1786805661.68 total 1045.19 last 70.98) - open/close 23060 (time 1786805671.69 total 1055.20 last 75.74) - open/close 23685 (time 1786805681.69 total 1065.20 last 62.50) - open/close 24346 (time 1786805691.71 total 1075.23 last 65.93) - open/close 25065 (time 1786805701.72 total 1085.23 last 71.87) - open/close 25737 (time 1786805711.72 total 1095.24 last 67.17) - open/close 26467 (time 1786805721.73 total 1105.24 last 72.95) - open/close 27165 (time 1786805731.73 total 1115.24 last 69.79) - open/close 27819 (time 1786805741.74 total 1125.26 last 65.32) - open/close 28567 (time 1786805751.76 total 1135.28 last 74.65) - open/close 29185 (time 1786805761.77 total 1145.28 last 61.79) - open/close 29952 (time 1786805771.77 total 1155.29 last 76.66) - open/close 30818 (time 1786805781.78 total 1165.29 last 86.54) - open/close 31604 (time 1786805791.79 total 1175.30 last 78.53) - open/close 32335 (time 1786805801.79 total 1185.30 last 73.08) - open/close 33112 (time 1786805811.79 total 1195.30 last 77.69) - open/close 33856 (time 1786805821.80 total 1205.31 last 74.37) - open/close 34680 (time 1786805831.82 total 1215.33 last 82.20) - open/close 35473 (time 1786805841.82 total 1225.34 last 79.26) - open/close 36132 (time 1786805851.83 total 1235.35 last 65.85) - open/close 36651 (time 1786805861.84 total 1245.35 last 51.87) - open/close 37258 (time 1786805871.85 total 1255.36 last 60.64) - open/close 37950 (time 1786805881.88 total 1265.39 last 68.99) - open/close 38685 (time 1786805891.89 total 1275.40 last 73.40) - open/close 39386 (time 1786805901.90 total 1285.41 last 70.07) - open/close 40000 (time 1786805910.05 total 1293.56 last 75.29) - open/close 40758 (time 1786805920.06 total 1303.57 last 75.74) - open/close 41584 (time 1786805930.06 total 1313.57 last 82.58) - open/close 42401 (time 1786805940.07 total 1323.58 last 81.63) - open/close 43189 (time 1786805950.08 total 1333.59 last 78.74) - open/close 43988 (time 1786805960.09 total 1343.60 last 79.80) - open/close 44751 (time 1786805970.09 total 1353.61 last 76.28) - open/close 45473 (time 1786805980.10 total 1363.62 last 72.12) - open/close 46298 (time 1786805990.11 total 1373.62 last 82.45) - open/close 47087 (time 1786806000.12 total 1383.63 last 78.86) - open/close 47829 (time 1786806010.12 total 1393.63 last 74.16) - open/close 48442 (time 1786806020.12 total 1403.64 last 61.28) - open/close 49150 (time 1786806030.13 total 1413.64 last 70.78) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1419.82 seconds: 34.96 ops/second - unlinked 0 (time 1786806041 ; total 0 ; last 0) - unlinked 10000 (time 1786806179 ; total 138 ; last 138) - unlinked 20000 (time 1786806324 ; total 283 ; last 145) - unlinked 30000 (time 1786806473 ; total 432 ; last 149) - unlinked 40000 (time 1786806623 ; total 582 ; last 150) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 722 seconds: 68.740997 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg411-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg411-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg411-server stop mds service on oleg411-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg411-server unloading modules via unload_modules_local on: 'oleg411-server' oleg411-server: oleg411-server.virtnet: executing unload_modules_local oleg411-server: modules unloaded. pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-server: ssh exited with exit code 2 pdsh@oleg411-client: oleg411-client: ssh exited with exit code 2