== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 18:55:27 (1785884127) start mds service on oleg213-server /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy Loading modules from /home/green/git/lustre-release/lustre detected 4 online CPUs by sysfs MODOPTS_LIBCFS= Force libcfs to create 2 CPU partitions ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' quota/lquota options: 'hash_lqs_cur_bits=3' mdt/mdt options: 'mdt_enable_flr_ec=1' ln: failed to create symbolic link '/sbin/.libs': Read-only file system loading modules on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing load_modules_local oleg213-server: Loading modules from /home/green/git/lustre-release/lustre oleg213-server: /home/green/git/lustre-release/lustre/tests/test-framework.sh: line 1048: echo: write error: Device or resource busy oleg213-server: detected 4 online CPUs by sysfs oleg213-server: MODOPTS_LIBCFS= oleg213-server: Force libcfs to create 2 CPU partitions oleg213-server: ptlrpc/ptlrpc options: 'lbug_on_grant_miscount=1' oleg213-server: quota/lquota options: 'hash_lqs_cur_bits=3' oleg213-server: mdt/mdt options: 'mdt_enable_flr_ec=1' Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg213-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg213-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg213-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg213-server: oleg213-server.virtnet: executing set_default_debug -1 all pdsh@oleg213-client: oleg213-server: ssh exited with exit code 1 Started lustre-OST0000 oleg213-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg213-client.virtnet: -o user_xattr,flock 192.168.202.113@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 155 (time 1785884296.00 total 10.34 last 14.99) - open/close 252 (time 1785884306.82 total 21.16 last 8.96) - open/close 346 (time 1785884316.86 total 31.19 last 9.37) - open/close 482 (time 1785884326.88 total 41.22 last 13.56) - open/close 622 (time 1785884337.48 total 51.82 last 13.21) - open/close 804 (time 1785884347.65 total 61.99 last 17.89) - open/close 883 (time 1785884358.10 total 72.44 last 7.56) - open/close 1035 (time 1785884368.37 total 82.71 last 14.80) - open/close 1143 (time 1785884379.56 total 93.90 last 9.65) - open/close 1361 (time 1785884390.12 total 104.46 last 20.65) - open/close 1545 (time 1785884400.19 total 114.53 last 18.26) - open/close 1725 (time 1785884410.95 total 125.28 last 16.74) - open/close 1842 (time 1785884421.00 total 135.34 last 11.63) - open/close 2007 (time 1785884431.69 total 146.03 last 15.43) - open/close 2153 (time 1785884442.15 total 156.48 last 13.97) - open/close 2275 (time 1785884452.49 total 166.83 last 11.79) - open/close 2476 (time 1785884462.50 total 176.83 last 20.09) - open/close 2681 (time 1785884472.97 total 187.31 last 19.56) - open/close 2816 (time 1785884483.99 total 198.33 last 12.25) - open/close 2984 (time 1785884494.01 total 208.34 last 16.78) - open/close 3097 (time 1785884504.11 total 218.44 last 11.19) - open/close 3238 (time 1785884514.96 total 229.30 last 12.99) - open/close 3380 (time 1785884525.72 total 240.06 last 13.20) - open/close 3509 (time 1785884535.74 total 250.08 last 12.87) - open/close 3630 (time 1785884546.37 total 260.71 last 11.38) - open/close 3739 (time 1785884556.58 total 270.92 last 10.68) - open/close 3849 (time 1785884566.64 total 280.98 last 10.93) - open/close 3998 (time 1785884576.84 total 291.17 last 14.62) - open/close 4095 (time 1785884588.05 total 302.39 last 8.65) - open/close 4238 (time 1785884598.60 total 312.94 last 13.56) - open/close 4331 (time 1785884608.61 total 322.95 last 9.29) - open/close 4489 (time 1785884619.65 total 333.98 last 14.32) - open/close 4582 (time 1785884629.84 total 344.18 last 9.12) - open/close 4736 (time 1785884640.85 total 355.19 last 13.99) - open/close 4868 (time 1785884651.65 total 365.99 last 12.22) - open/close 4991 (time 1785884661.72 total 376.06 last 12.21) - open/close 5124 (time 1785884671.75 total 386.09 last 13.26) - open/close 5271 (time 1785884681.78 total 396.12 last 14.66) - open/close 5395 (time 1785884692.61 total 406.95 last 11.45) - open/close 5556 (time 1785884702.62 total 416.96 last 16.09) - open/close 5688 (time 1785884713.61 total 427.95 last 12.01) - open/close 5858 (time 1785884723.65 total 437.98 last 16.94) - open/close 6024 (time 1785884733.65 total 447.99 last 16.60) - open/close 6124 (time 1785884744.13 total 458.47 last 9.54) - open/close 6268 (time 1785884755.21 total 469.54 last 13.01) - open/close 6375 (time 1785884765.95 total 480.29 last 9.96) - open/close 6498 (time 1785884776.67 total 491.00 last 11.48) - open/close 6597 (time 1785884786.67 total 501.01 last 9.89) - open/close 6717 (time 1785884796.93 total 511.26 last 11.70) - open/close 6824 (time 1785884806.95 total 521.29 last 10.68) - open/close 6926 (time 1785884817.85 total 532.19 last 9.36) - open/close 7023 (time 1785884828.56 total 542.90 last 9.06) - open/close 7150 (time 1785884838.67 total 553.01 last 12.57) - open/close 7376 (time 1785884848.83 total 563.16 last 22.25) - open/close 7484 (time 1785884859.28 total 573.62 last 10.33) - open/close 7674 (time 1785884869.29 total 583.63 last 18.98) - open/close 7840 (time 1785884879.43 total 593.77 last 16.36) - open/close 7952 (time 1785884889.67 total 604.01 last 10.94) - open/close 8134 (time 1785884900.05 total 614.38 last 17.55) - open/close 8235 (time 1785884910.16 total 624.49 last 9.99) - open/close 8360 (time 1785884920.16 total 634.50 last 12.49) - open/close 8559 (time 1785884930.17 total 644.51 last 19.89) - open/close 8695 (time 1785884941.26 total 655.60 last 12.26) - open/close 8846 (time 1785884951.58 total 665.92 last 14.63) - open/close 8971 (time 1785884961.59 total 675.92 last 12.50) - open/close 9150 (time 1785884972.53 total 686.87 last 16.35) - open/close 9301 (time 1785884983.49 total 697.83 last 13.78) - open/close 9457 (time 1785884993.87 total 708.21 last 15.03) - open/close 9597 (time 1785885004.43 total 718.76 last 13.26) - open/close 9759 (time 1785885014.44 total 728.78 last 16.17) - open/close 9864 (time 1785885024.53 total 738.87 last 10.41) - open/close 10000 (time 1785885034.34 total 748.67 last 13.86) - open/close 10129 (time 1785885044.39 total 758.72 last 12.84) - open/close 10237 (time 1785885054.41 total 768.75 last 10.78) - open/close 10374 (time 1785885065.27 total 779.61 last 12.61) - open/close 10507 (time 1785885075.34 total 789.67 last 13.22) - open/close 10615 (time 1785885085.47 total 799.81 last 10.65) - open/close 10741 (time 1785885096.33 total 810.67 last 11.60) - open/close 10870 (time 1785885107.15 total 821.49 last 11.93) - open/close 10990 (time 1785885117.58 total 831.91 last 11.51) - open/close 11115 (time 1785885127.60 total 841.93 last 12.47) - open/close 11215 (time 1785885138.11 total 852.45 last 9.51) - open/close 11335 (time 1785885148.89 total 863.23 last 11.13) - open/close 11420 (time 1785885158.93 total 873.27 last 8.47) - open/close 11581 (time 1785885171.91 total 886.25 last 12.41) - open/close 11701 (time 1785885182.52 total 896.85 last 11.32) - open/close 11834 (time 1785885192.54 total 906.88 last 13.27) - open/close 11932 (time 1785885202.58 total 916.91 last 9.76) - open/close 12026 (time 1785885212.67 total 927.01 last 9.31) - open/close 12124 (time 1785885223.62 total 937.96 last 8.95) - open/close 12279 (time 1785885234.14 total 948.48 last 14.73) - open/close 12399 (time 1785885244.82 total 959.16 last 11.24) - open/close 12573 (time 1785885255.41 total 969.75 last 16.44) - open/close 12744 (time 1785885265.89 total 980.23 last 16.31) - open/close 12844 (time 1785885275.93 total 990.26 last 9.97) - open/close 12964 (time 1785885286.92 total 1001.26 last 10.91) - open/close 13098 (time 1785885297.89 total 1012.23 last 12.22) - open/close 13267 (time 1785885308.43 total 1022.77 last 16.04) - open/close 13404 (time 1785885318.66 total 1033.00 last 13.38) - open/close 13550 (time 1785885328.67 total 1043.01 last 14.59) - open/close 13694 (time 1785885339.14 total 1053.48 last 13.76) - open/close 13789 (time 1785885349.44 total 1063.78 last 9.22) - open/close 13953 (time 1785885359.45 total 1073.78 last 16.39) - open/close 14068 (time 1785885369.55 total 1083.89 last 11.38) - open/close 14162 (time 1785885379.57 total 1093.90 last 9.38) - open/close 14262 (time 1785885389.58 total 1103.91 last 9.99) - open/close 14355 (time 1785885399.61 total 1113.95 last 9.27) - open/close 14482 (time 1785885410.28 total 1124.62 last 11.90) - open/close 14673 (time 1785885420.31 total 1134.65 last 19.04) - open/close 14821 (time 1785885431.39 total 1145.73 last 13.35) - open/close 14958 (time 1785885441.57 total 1155.91 last 13.46) - open/close 15050 (time 1785885451.57 total 1165.91 last 9.20) - open/close 15153 (time 1785885461.59 total 1175.93 last 10.28) - open/close 15313 (time 1785885472.27 total 1186.61 last 14.98) - open/close 15434 (time 1785885482.35 total 1196.69 last 12.01) - open/close 15643 (time 1785885492.51 total 1206.85 last 20.57) - open/close 15747 (time 1785885502.86 total 1217.20 last 10.05) - open/close 15840 (time 1785885512.88 total 1227.21 last 9.29) - open/close 15918 (time 1785885523.35 total 1237.69 last 7.45) - open/close 16059 (time 1785885533.59 total 1247.93 last 13.76) - open/close 16176 (time 1785885544.06 total 1258.40 last 11.18) - open/close 16322 (time 1785885554.07 total 1268.41 last 14.58) - open/close 16465 (time 1785885565.12 total 1279.46 last 12.95) - open/close 16570 (time 1785885575.99 total 1290.33 last 9.65) - open/close 16701 (time 1785885586.01 total 1300.35 last 13.08) - open/close 16819 (time 1785885596.60 total 1310.94 last 11.14) - open/close 16965 (time 1785885607.18 total 1321.52 last 13.80) - open/close 17106 (time 1785885618.20 total 1332.54 last 12.79) - open/close 17177 (time 1785885628.83 total 1343.17 last 6.68) - open/close 17323 (time 1785885638.84 total 1353.18 last 14.58) - open/close 17389 (time 1785885648.88 total 1363.22 last 6.57) - open/close 17518 (time 1785885658.90 total 1373.24 last 12.87) - open/close 17674 (time 1785885668.94 total 1383.28 last 15.54) - open/close 17848 (time 1785885679.76 total 1394.10 last 16.08) - open/close 17946 (time 1785885690.19 total 1404.53 last 9.40) - open/close 18064 (time 1785885700.94 total 1415.28 last 10.98) - open/close 18172 (time 1785885711.44 total 1425.78 last 10.28) - open/close 18271 (time 1785885722.38 total 1436.72 last 9.05) - open/close 18342 (time 1785885732.51 total 1446.85 last 7.01) - open/close 18440 (time 1785885742.53 total 1456.86 last 9.79) - open/close 18527 (time 1785885754.20 total 1468.54 last 7.45) - open/close 19361 (time 1785885764.21 total 1478.55 last 83.34) - open/close 19874 (time 1785885774.23 total 1488.57 last 51.21) - open/close 20000 (time 1785885776.97 total 1491.31 last 45.92) - open/close 20540 (time 1785885786.97 total 1501.31 last 53.99) - open/close 20982 (time 1785885797.00 total 1511.33 last 44.10) - open/close 21425 (time 1785885807.01 total 1521.34 last 44.25) - open/close 21906 (time 1785885817.02 total 1531.35 last 48.06) - open/close 22455 (time 1785885827.05 total 1541.38 last 54.73) - open/close 23084 (time 1785885837.05 total 1551.39 last 62.85) - open/close 23860 (time 1785885847.07 total 1561.41 last 77.47) - open/close 24402 (time 1785885857.07 total 1571.41 last 54.19) - open/close 24927 (time 1785885867.09 total 1581.43 last 52.39) - open/close 25415 (time 1785885877.10 total 1591.44 last 48.77) - open/close 25918 (time 1785885887.10 total 1601.44 last 50.30) - open/close 26397 (time 1785885897.10 total 1611.44 last 47.88) - open/close 26936 (time 1785885907.11 total 1621.45 last 53.85) - open/close 27622 (time 1785885917.12 total 1631.45 last 68.57) - open/close 28109 (time 1785885927.12 total 1641.46 last 48.69) - open/close 28651 (time 1785885937.14 total 1651.48 last 54.08) - open/close 29077 (time 1785885947.16 total 1661.50 last 42.51) - open/close 29391 (time 1785885957.23 total 1671.57 last 31.17) - open/close 29761 (time 1785885967.25 total 1681.59 last 36.93) - open/close 30000 (time 1785885973.90 total 1688.24 last 35.94) - open/close 30422 (time 1785885983.92 total 1698.26 last 42.11) - open/close 30880 (time 1785885993.93 total 1708.26 last 45.79) - open/close 31469 (time 1785886003.93 total 1718.27 last 58.85) - open/close 32128 (time 1785886013.97 total 1728.31 last 65.67) - open/close 32768 (time 1785886023.99 total 1738.33 last 63.87) - open/close 33249 (time 1785886033.99 total 1748.33 last 48.10) - open/close 33936 (time 1785886043.99 total 1758.33 last 68.69) - open/close 34547 (time 1785886054.00 total 1768.34 last 61.01) - open/close 34892 (time 1785886066.27 total 1780.60 last 28.13) - open/close 35189 (time 1785886076.27 total 1790.61 last 29.68) - open/close 35459 (time 1785886086.51 total 1800.84 last 26.38) - open/close 35716 (time 1785886096.53 total 1810.86 last 25.65) - open/close 36112 (time 1785886106.53 total 1820.87 last 39.57) - open/close 36544 (time 1785886116.54 total 1830.88 last 43.17) - open/close 37005 (time 1785886126.55 total 1840.88 last 46.08) - open/close 37418 (time 1785886136.56 total 1850.89 last 41.26) - open/close 37894 (time 1785886146.56 total 1860.90 last 47.56) - open/close 38405 (time 1785886156.59 total 1870.93 last 50.98) - open/close 38881 (time 1785886166.59 total 1880.93 last 47.57) - open/close 39385 (time 1785886176.61 total 1890.95 last 50.30) - open/close 39948 (time 1785886186.63 total 1900.97 last 56.20) - open/close 40442 (time 1785886196.64 total 1910.97 last 49.38) - open/close 41038 (time 1785886206.65 total 1920.98 last 59.54) - open/close 41553 (time 1785886216.66 total 1931.00 last 51.43) - open/close 42021 (time 1785886226.67 total 1941.01 last 46.74) - open/close 42502 (time 1785886236.68 total 1951.02 last 48.06) - open/close 43167 (time 1785886246.69 total 1961.02 last 66.47) - open/close 43791 (time 1785886256.69 total 1971.03 last 62.39) - open/close 44346 (time 1785886266.71 total 1981.04 last 55.40) - open/close 45091 (time 1785886276.71 total 1991.05 last 74.47) - open/close 45557 (time 1785886286.71 total 2001.05 last 46.60) - open/close 46066 (time 1785886296.71 total 2011.05 last 50.89) - open/close 46562 (time 1785886306.72 total 2021.06 last 49.56) - open/close 47179 (time 1785886316.73 total 2031.07 last 61.64) - open/close 47925 (time 1785886326.75 total 2041.09 last 74.47) - open/close 48408 (time 1785886336.76 total 2051.10 last 48.23) - open/close 48904 (time 1785886346.77 total 2061.11 last 49.57) - open/close 49320 (time 1785886356.77 total 2071.11 last 41.57) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No space left on device total: 49631 open/close in 2078.64 seconds: 23.88 ops/second - unlinked 0 (time 1785886370 ; total 0 ; last 0) - unlinked 10000 (time 1785886562 ; total 192 ; last 192) - unlinked 20000 (time 1785886755 ; total 385 ; last 193) - unlinked 30000 (time 1785886956 ; total 586 ; last 201) - unlinked 40000 (time 1785887134 ; total 764 ; last 178) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49631) error: No such file or directory total: 49631 unlinks in 958 seconds: 51.806889 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg213-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg213-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg213-server stop mds service on oleg213-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg213-server unloading modules via unload_modules_local on: 'oleg213-server' oleg213-server: oleg213-server.virtnet: executing unload_modules_local oleg213-server: modules unloaded. pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-server: ssh exited with exit code 2 pdsh@oleg213-client: oleg213-client: ssh exited with exit code 2