== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 04:35:53 (1779698153) start mds service on oleg435-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing load_modules_local oleg435-server: Loading modules from /home/green/git/lustre-release/lustre oleg435-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1042: echo: write error: Device or resource busy oleg435-server: detected 4 online CPUs by sysfs oleg435-server: MODOPTS_LIBCFS= oleg435-server: Force libcfs to create 2 CPU partitions oleg435-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg435-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg435-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg435-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg435-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg435-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg435-server: oleg435-server.virtnet: executing set_default_debug -1 all pdsh@oleg435-client: oleg435-server: ssh exited with exit code 1 Started lustre-OST0000 oleg435-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg435-client.virtnet: -o user_xattr,flock 192.168.204.135@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 113 (time 1779698272.73 total 10.01 last 11.29) - open/close 226 (time 1779698282.76 total 20.04 last 11.26) - open/close 305 (time 1779698293.28 total 30.55 last 7.52) - open/close 406 (time 1779698304.60 total 41.88 last 8.92) - open/close 585 (time 1779698315.47 total 52.75 last 16.47) - open/close 834 (time 1779698325.47 total 62.75 last 24.89) - open/close 1069 (time 1779698335.48 total 72.76 last 23.49) - open/close 1138 (time 1779698346.49 total 83.77 last 6.27) - open/close 1203 (time 1779698356.63 total 93.91 last 6.41) - open/close 1369 (time 1779698367.45 total 104.73 last 15.34) - open/close 1444 (time 1779698378.48 total 115.75 last 6.80) - open/close 1623 (time 1779698388.84 total 126.12 last 17.27) - open/close 1739 (time 1779698401.26 total 138.54 last 9.34) - open/close 1829 (time 1779698411.79 total 149.06 last 8.55) - open/close 1917 (time 1779698421.99 total 159.27 last 8.62) - open/close 1999 (time 1779698432.00 total 169.27 last 8.20) - open/close 2241 (time 1779698443.04 total 180.32 last 21.91) - open/close 2430 (time 1779698453.04 total 190.32 last 18.90) - open/close 2561 (time 1779698463.07 total 200.35 last 13.06) - open/close 2776 (time 1779698473.44 total 210.72 last 20.73) - open/close 2977 (time 1779698483.45 total 220.72 last 20.09) - open/close 3197 (time 1779698493.74 total 231.01 last 21.38) - open/close 3415 (time 1779698504.27 total 241.55 last 20.69) - open/close 3632 (time 1779698515.99 total 253.27 last 18.52) - open/close 3757 (time 1779698526.03 total 263.30 last 12.45) - open/close 3938 (time 1779698536.22 total 273.50 last 17.75) - open/close 4135 (time 1779698546.25 total 283.52 last 19.66) - open/close 4312 (time 1779698556.26 total 293.53 last 17.68) - open/close 4432 (time 1779698566.99 total 304.27 last 11.18) - open/close 4570 (time 1779698577.01 total 314.28 last 13.78) - open/close 4802 (time 1779698587.01 total 324.28 last 23.20) - open/close 4961 (time 1779698597.78 total 335.06 last 14.76) - open/close 5037 (time 1779698609.02 total 346.29 last 6.76) - open/close 5208 (time 1779698619.02 total 356.30 last 17.10) - open/close 5328 (time 1779698629.84 total 367.11 last 11.09) - open/close 5485 (time 1779698640.39 total 377.67 last 14.87) - open/close 5682 (time 1779698650.40 total 387.67 last 19.69) - open/close 5831 (time 1779698660.41 total 397.69 last 14.87) - open/close 5993 (time 1779698670.43 total 407.71 last 16.17) - open/close 6121 (time 1779698681.44 total 418.72 last 11.63) - open/close 6286 (time 1779698691.94 total 429.21 last 15.72) - open/close 6354 (time 1779698702.93 total 440.20 last 6.19) - open/close 6529 (time 1779698713.68 total 450.95 last 16.28) - open/close 6599 (time 1779698723.77 total 461.04 last 6.94) - open/close 6776 (time 1779698733.77 total 471.05 last 17.69) - open/close 6926 (time 1779698744.43 total 481.71 last 14.07) - open/close 7142 (time 1779698755.36 total 492.64 last 19.76) - open/close 7199 (time 1779698765.95 total 503.23 last 5.38) - open/close 7289 (time 1779698776.08 total 513.36 last 8.89) - open/close 7393 (time 1779698786.21 total 523.49 last 10.27) - open/close 7477 (time 1779698796.45 total 533.73 last 8.20) - open/close 7645 (time 1779698806.48 total 543.75 last 16.75) - open/close 7720 (time 1779698821.65 total 558.93 last 4.94) - open/close 7898 (time 1779698831.66 total 568.94 last 17.79) - open/close 7994 (time 1779698841.75 total 579.03 last 9.52) - open/close 8084 (time 1779698851.76 total 589.04 last 8.99) - open/close 8263 (time 1779698861.77 total 599.04 last 17.89) - open/close 8345 (time 1779698871.96 total 609.24 last 8.04) - open/close 8491 (time 1779698882.85 total 620.13 last 13.41) - open/close 8579 (time 1779698893.55 total 630.82 last 8.23) - open/close 8698 (time 1779698903.91 total 641.19 last 11.48) - open/close 8788 (time 1779698914.27 total 651.55 last 8.69) - open/close 8829 (time 1779698924.83 total 662.11 last 3.88) - open/close 8960 (time 1779698935.18 total 672.46 last 12.65) - open/close 9144 (time 1779698946.07 total 683.35 last 16.90) - open/close 9310 (time 1779698956.85 total 694.12 last 15.40) - open/close 9514 (time 1779698967.50 total 704.78 last 19.15) - open/close 9701 (time 1779698978.26 total 715.54 last 17.37) - open/close 9817 (time 1779698989.10 total 726.37 last 10.71) - open/close 9923 (time 1779698999.85 total 737.12 last 9.86) - open/close 10000 (time 1779699007.68 total 744.96 last 9.83) - open/close 10125 (time 1779699018.64 total 755.91 last 11.41) - open/close 10251 (time 1779699028.72 total 765.99 last 12.50) - open/close 10410 (time 1779699038.89 total 776.17 last 15.63) - open/close 10541 (time 1779699048.90 total 786.17 last 13.09) - open/close 10659 (time 1779699059.48 total 796.75 last 11.15) - open/close 10845 (time 1779699070.11 total 807.38 last 17.50) - open/close 11001 (time 1779699080.11 total 817.39 last 15.60) - open/close 11102 (time 1779699090.27 total 827.55 last 9.94) - open/close 11268 (time 1779699100.99 total 838.27 last 15.49) - open/close 11435 (time 1779699113.00 total 850.27 last 13.91) - open/close 11586 (time 1779699123.38 total 860.66 last 14.54) - open/close 11769 (time 1779699133.47 total 870.75 last 18.14) - open/close 11890 (time 1779699144.35 total 881.63 last 11.12) - open/close 12049 (time 1779699154.83 total 892.11 last 15.17) - open/close 12218 (time 1779699165.15 total 902.43 last 16.38) - open/close 12432 (time 1779699175.17 total 912.44 last 21.36) - open/close 12547 (time 1779699185.18 total 922.45 last 11.49) - open/close 12639 (time 1779699195.19 total 932.47 last 9.19) - open/close 12797 (time 1779699207.13 total 944.41 last 13.24) - open/close 12946 (time 1779699217.62 total 954.89 last 14.21) - open/close 13081 (time 1779699227.63 total 964.91 last 13.48) - open/close 13337 (time 1779699237.66 total 974.94 last 25.53) - open/close 13564 (time 1779699248.42 total 985.69 last 21.10) - open/close 13750 (time 1779699258.43 total 995.70 last 18.59) - open/close 13937 (time 1779699268.84 total 1006.11 last 17.96) - open/close 14010 (time 1779699279.78 total 1017.05 last 6.67) - open/close 14263 (time 1779699289.83 total 1027.11 last 25.17) - open/close 14408 (time 1779699300.50 total 1037.78 last 13.59) - open/close 14612 (time 1779699311.10 total 1048.38 last 19.25) - open/close 14752 (time 1779699322.58 total 1059.85 last 12.20) - open/close 14884 (time 1779699332.83 total 1070.10 last 12.87) - open/close 15057 (time 1779699344.33 total 1081.61 last 15.03) - open/close 15192 (time 1779699354.41 total 1091.68 last 13.40) - open/close 15579 (time 1779699364.42 total 1101.70 last 38.64) - open/close 16120 (time 1779699374.43 total 1111.71 last 54.07) - open/close 16627 (time 1779699384.48 total 1121.76 last 50.42) - open/close 17316 (time 1779699394.50 total 1131.77 last 68.81) - open/close 18013 (time 1779699404.50 total 1141.77 last 69.69) - open/close 18699 (time 1779699414.50 total 1151.78 last 68.58) - open/close 19354 (time 1779699424.51 total 1161.78 last 65.47) - open/close 20000 (time 1779699434.15 total 1171.43 last 66.99) - open/close 20645 (time 1779699444.16 total 1181.44 last 64.41) - open/close 21231 (time 1779699454.17 total 1191.45 last 58.56) - open/close 21890 (time 1779699464.18 total 1201.46 last 65.83) - open/close 22500 (time 1779699474.18 total 1211.46 last 60.98) - open/close 23191 (time 1779699484.18 total 1221.46 last 69.10) - open/close 23889 (time 1779699494.20 total 1231.48 last 69.70) - open/close 24576 (time 1779699504.22 total 1241.49 last 68.58) - open/close 25259 (time 1779699514.22 total 1251.50 last 68.28) - open/close 25947 (time 1779699524.24 total 1261.52 last 68.66) - open/close 26691 (time 1779699534.24 total 1271.52 last 74.40) - open/close 27392 (time 1779699544.25 total 1281.53 last 70.00) - open/close 28083 (time 1779699554.26 total 1291.53 last 69.09) - open/close 28701 (time 1779699564.26 total 1301.54 last 61.74) - open/close 29300 (time 1779699574.29 total 1311.57 last 59.73) - open/close 30000 (time 1779699584.15 total 1321.43 last 71.00) - open/close 30742 (time 1779699594.16 total 1331.44 last 74.15) - open/close 31466 (time 1779699604.17 total 1341.45 last 72.29) - open/close 32273 (time 1779699614.19 total 1351.47 last 80.56) - open/close 33048 (time 1779699624.19 total 1361.47 last 77.48) - open/close 33713 (time 1779699634.20 total 1371.47 last 66.49) - open/close 34429 (time 1779699644.20 total 1381.48 last 71.57) - open/close 35102 (time 1779699654.20 total 1391.48 last 67.29) - open/close 35705 (time 1779699664.21 total 1401.49 last 60.27) - open/close 36359 (time 1779699674.22 total 1411.49 last 65.34) - open/close 37002 (time 1779699684.22 total 1421.50 last 64.29) - open/close 37596 (time 1779699694.22 total 1431.50 last 59.40) - open/close 38168 (time 1779699704.22 total 1441.50 last 57.18) - open/close 38885 (time 1779699714.23 total 1451.51 last 71.63) - open/close 39459 (time 1779699724.24 total 1461.52 last 57.35) - open/close 40000 (time 1779699732.06 total 1469.34 last 69.18) - open/close 40695 (time 1779699742.09 total 1479.37 last 69.28) - open/close 41436 (time 1779699752.10 total 1489.38 last 74.04) - open/close 42141 (time 1779699762.12 total 1499.39 last 70.38) - open/close 42805 (time 1779699772.13 total 1509.41 last 66.30) - open/close 43468 (time 1779699782.13 total 1519.41 last 66.28) - open/close 44233 (time 1779699792.15 total 1529.43 last 76.36) - open/close 44931 (time 1779699802.17 total 1539.44 last 69.71) - open/close 45652 (time 1779699812.18 total 1549.45 last 72.03) - open/close 46273 (time 1779699822.19 total 1559.47 last 61.98) - open/close 46876 (time 1779699832.22 total 1569.49 last 60.17) - open/close 47484 (time 1779699842.23 total 1579.50 last 60.75) - open/close 48116 (time 1779699852.23 total 1589.51 last 63.15) - open/close 48730 (time 1779699862.25 total 1599.53 last 61.31) - open/close 49370 (time 1779699872.29 total 1609.56 last 63.76) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 1613.72 seconds: 30.76 ops/second - unlinked 0 (time 1779699881 ; total 0 ; last 0) - unlinked 10000 (time 1779700033 ; total 152 ; last 152) - unlinked 20000 (time 1779700185 ; total 304 ; last 152) - unlinked 30000 (time 1779700334 ; total 453 ; last 149) - unlinked 40000 (time 1779700494 ; total 613 ; last 160) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 752 seconds: 65.998672 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg435-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg435-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg435-server stop mds service on oleg435-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg435-server unloading modules via unload_modules_local on: 'oleg435-server' oleg435-server: oleg435-server.virtnet: executing unload_modules_local oleg435-server: modules unloaded. pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-client: ssh exited with exit code 2 pdsh@oleg435-client: oleg435-server: ssh exited with exit code 2