== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:43 (1709786803) start mds service on oleg311-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg311-server Starting mds2: -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg311-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg311-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg311-server: oleg311-server.virtnet: executing set_default_debug -1 all pdsh@oleg311-client: oleg311-server: ssh exited with exit code 1 Started lustre-OST0000 oleg311-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg311-client.virtnet: -o user_xattr,flock oleg311-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 286 (time 1709786846.70 total 10.31 last 27.75) - open/close 576 (time 1709786857.43 total 21.03 last 27.04) - open/close 845 (time 1709786867.62 total 31.22 last 26.39) - open/close 1162 (time 1709786877.63 total 41.24 last 31.66) - open/close 1378 (time 1709786888.15 total 51.75 last 20.54) - open/close 1538 (time 1709786898.61 total 62.22 last 15.29) - open/close 1726 (time 1709786908.92 total 72.53 last 18.24) - open/close 1981 (time 1709786919.58 total 83.19 last 23.93) - open/close 2184 (time 1709786929.99 total 93.60 last 19.49) - open/close 2366 (time 1709786940.80 total 104.41 last 16.84) - open/close 2614 (time 1709786950.80 total 114.41 last 24.80) - open/close 2802 (time 1709786961.06 total 124.67 last 18.32) - open/close 3001 (time 1709786971.90 total 135.50 last 18.37) - open/close 3170 (time 1709786982.33 total 145.94 last 16.20) - open/close 3333 (time 1709786993.38 total 156.98 last 14.75) - open/close 3464 (time 1709787003.39 total 166.99 last 13.08) - open/close 3761 (time 1709787013.71 total 177.32 last 28.76) - open/close 3907 (time 1709787023.72 total 187.33 last 14.59) - open/close 4049 (time 1709787034.68 total 198.29 last 12.96) - open/close 4244 (time 1709787044.87 total 208.48 last 19.13) - open/close 4458 (time 1709787055.58 total 219.19 last 19.98) - open/close 4623 (time 1709787066.03 total 229.64 last 15.79) - open/close 4920 (time 1709787076.17 total 239.77 last 29.31) - open/close 5172 (time 1709787086.40 total 250.01 last 24.62) - open/close 5338 (time 1709787096.50 total 260.11 last 16.43) - open/close 5506 (time 1709787106.52 total 270.13 last 16.76) - open/close 5648 (time 1709787117.29 total 280.90 last 13.19) - open/close 5828 (time 1709787127.36 total 290.97 last 17.87) - open/close 5983 (time 1709787137.37 total 300.98 last 15.49) - open/close 6227 (time 1709787148.27 total 311.88 last 22.37) - open/close 6480 (time 1709787158.28 total 321.89 last 25.28) - open/close 6709 (time 1709787169.21 total 332.81 last 20.96) - open/close 6856 (time 1709787180.22 total 343.83 last 13.34) - open/close 7073 (time 1709787190.22 total 353.83 last 21.70) - open/close 7236 (time 1709787200.60 total 364.21 last 15.71) - open/close 7411 (time 1709787211.69 total 375.29 last 15.78) - open/close 7581 (time 1709787221.69 total 385.30 last 16.99) - open/close 7811 (time 1709787232.71 total 396.32 last 20.87) - open/close 7969 (time 1709787243.68 total 407.29 last 14.40) - open/close 8153 (time 1709787253.68 total 417.29 last 18.40) - open/close 8315 (time 1709787264.34 total 427.95 last 15.20) - open/close 8533 (time 1709787274.36 total 437.97 last 21.77) - open/close 8673 (time 1709787284.45 total 448.06 last 13.87) - open/close 8851 (time 1709787295.31 total 458.92 last 16.40) - open/close 9058 (time 1709787305.91 total 469.52 last 19.52) - open/close 9210 (time 1709787316.22 total 479.82 last 14.75) - open/close 9344 (time 1709787326.48 total 490.09 last 13.06) - open/close 9461 (time 1709787337.24 total 500.85 last 10.87) - open/close 9641 (time 1709787347.99 total 511.60 last 16.74) - open/close 9806 (time 1709787358.01 total 521.61 last 16.48) - open/close 9925 (time 1709787368.02 total 531.63 last 11.88) - open/close 10000 (time 1709787370.53 total 534.13 last 29.98) - open/close 10479 (time 1709787380.53 total 544.14 last 47.87) - open/close 10646 (time 1709787395.12 total 558.73 last 11.45) - open/close 11114 (time 1709787405.15 total 568.75 last 46.70) - open/close 11543 (time 1709787415.16 total 578.77 last 42.82) - open/close 11982 (time 1709787425.22 total 588.82 last 43.67) - open/close 12447 (time 1709787435.25 total 598.86 last 46.34) - open/close 12924 (time 1709787445.26 total 608.87 last 47.65) - open/close 13401 (time 1709787455.28 total 618.88 last 47.64) - open/close 13815 (time 1709787465.29 total 628.89 last 41.35) - open/close 14202 (time 1709787475.31 total 638.91 last 38.62) - open/close 14647 (time 1709787485.32 total 648.92 last 44.45) - open/close 15031 (time 1709787495.33 total 658.94 last 38.36) - open/close 15428 (time 1709787505.34 total 668.95 last 39.65) - open/close 15792 (time 1709787515.35 total 678.95 last 36.39) - open/close 16163 (time 1709787525.36 total 688.96 last 37.06) - open/close 16369 (time 1709787543.11 total 706.71 last 11.61) - open/close 16553 (time 1709787559.66 total 723.27 last 11.11) - open/close 16856 (time 1709787569.69 total 733.30 last 30.21) - open/close 17181 (time 1709787579.70 total 743.31 last 32.48) - open/close 17490 (time 1709787589.70 total 753.31 last 30.89) - open/close 17769 (time 1709787599.71 total 763.32 last 27.87) - open/close 18074 (time 1709787609.74 total 773.35 last 30.41) - open/close 18238 (time 1709787625.58 total 789.19 last 10.36) - open/close 18570 (time 1709787635.60 total 799.21 last 33.13) - open/close 18858 (time 1709787645.60 total 809.21 last 28.80) - open/close 19169 (time 1709787655.67 total 819.28 last 30.89) - open/close 19498 (time 1709787665.71 total 829.32 last 32.76) - open/close 19761 (time 1709787675.76 total 839.37 last 26.18) - open/close 20000 (time 1709787684.64 total 848.24 last 26.93) - open/close 20258 (time 1709787694.69 total 858.30 last 25.65) - open/close 20471 (time 1709787704.70 total 868.31 last 21.29) - open/close 20704 (time 1709787714.71 total 878.32 last 23.28) - open/close 20975 (time 1709787724.72 total 888.33 last 27.06) - open/close 21271 (time 1709787734.78 total 898.38 last 29.44) - open/close 21527 (time 1709787744.78 total 908.39 last 25.60) - open/close 21774 (time 1709787754.80 total 918.40 last 24.66) - open/close 22050 (time 1709787764.80 total 928.41 last 27.58) - open/close 22315 (time 1709787774.83 total 938.44 last 26.42) - open/close 22593 (time 1709787784.85 total 948.46 last 27.76) - open/close 22861 (time 1709787794.87 total 958.48 last 26.74) - open/close 22941 (time 1709787808.50 total 972.11 last 5.87) - open/close 23231 (time 1709787818.52 total 982.13 last 28.95) - open/close 23472 (time 1709787838.13 total 1001.74 last 12.29) - open/close 23782 (time 1709787848.14 total 1011.74 last 30.98) - open/close 24120 (time 1709787858.14 total 1021.74 last 33.80) - open/close 24447 (time 1709787868.18 total 1031.79 last 32.55) - open/close 24744 (time 1709787878.22 total 1041.83 last 29.57) - open/close 25053 (time 1709787888.28 total 1051.89 last 30.72) - open/close 25359 (time 1709787898.29 total 1061.90 last 30.57) - open/close 25688 (time 1709787908.29 total 1071.90 last 32.90) - open/close 25970 (time 1709787929.05 total 1092.66 last 13.58) - open/close 25997 (time 1709787940.98 total 1104.59 last 2.26) - open/close 26273 (time 1709787951.03 total 1114.63 last 27.48) - open/close 26573 (time 1709787961.04 total 1124.65 last 29.96) - open/close 26887 (time 1709787971.06 total 1134.66 last 31.35) - open/close 27182 (time 1709787981.07 total 1144.68 last 29.46) - open/close 27513 (time 1709787991.09 total 1154.70 last 33.03) - open/close 27843 (time 1709788001.11 total 1164.72 last 32.93) - open/close 28153 (time 1709788011.13 total 1174.74 last 30.95) - open/close 28475 (time 1709788021.15 total 1184.76 last 32.13) - open/close 28764 (time 1709788031.15 total 1194.76 last 28.90) - open/close 29071 (time 1709788041.16 total 1204.77 last 30.67) - open/close 29370 (time 1709788051.19 total 1214.79 last 29.83) - open/close 29664 (time 1709788061.20 total 1224.81 last 29.35) - open/close 29916 (time 1709788071.23 total 1234.83 last 25.14) - open/close 29927 (time 1709788082.59 total 1246.20 last 0.97) - open/close 30303 (time 1709788092.60 total 1256.20 last 37.57) - open/close 30679 (time 1709788102.62 total 1266.22 last 37.53) - open/close 31043 (time 1709788112.64 total 1276.25 last 36.30) - open/close 31401 (time 1709788122.66 total 1286.27 last 35.75) - open/close 31771 (time 1709788132.67 total 1296.28 last 36.96) - open/close 32138 (time 1709788142.69 total 1306.29 last 36.64) - open/close 32500 (time 1709788152.71 total 1316.31 last 36.13) - open/close 32863 (time 1709788162.72 total 1326.33 last 36.25) - open/close 33214 (time 1709788172.74 total 1336.34 last 35.04) - open/close 33551 (time 1709788182.75 total 1346.36 last 33.64) - open/close 33903 (time 1709788192.77 total 1356.38 last 35.14) - open/close 34094 (time 1709788209.56 total 1373.17 last 11.37) - open/close 34431 (time 1709788219.59 total 1383.20 last 33.61) - open/close 34771 (time 1709788229.62 total 1393.22 last 33.91) - open/close 35096 (time 1709788239.62 total 1403.22 last 32.50) - open/close 35422 (time 1709788249.64 total 1413.24 last 32.54) - open/close 35733 (time 1709788259.64 total 1423.25 last 31.08) - open/close 36040 (time 1709788269.66 total 1433.27 last 30.63) - open/close 36368 (time 1709788279.68 total 1443.29 last 32.74) - open/close 36676 (time 1709788289.70 total 1453.31 last 30.75) - open/close 36975 (time 1709788299.72 total 1463.32 last 29.85) - open/close 37266 (time 1709788309.72 total 1473.33 last 29.10) - open/close 37574 (time 1709788319.74 total 1483.35 last 30.72) - open/close 37882 (time 1709788329.76 total 1493.36 last 30.76) - open/close 38192 (time 1709788339.78 total 1503.39 last 30.93) - open/close 38508 (time 1709788349.80 total 1513.41 last 31.52) - open/close 38829 (time 1709788359.80 total 1523.41 last 32.10) - open/close 39121 (time 1709788369.83 total 1533.44 last 29.11) - open/close 39426 (time 1709788379.87 total 1543.48 last 30.38) - open/close 39722 (time 1709788389.90 total 1553.51 last 29.52) - open/close 40000 (time 1709788399.05 total 1562.66 last 30.37) - open/close 40300 (time 1709788409.05 total 1572.66 last 30.00) - open/close 40596 (time 1709788419.08 total 1582.69 last 29.52) - open/close 40887 (time 1709788429.10 total 1592.71 last 29.03) - open/close 41198 (time 1709788439.11 total 1602.72 last 31.09) - open/close 41489 (time 1709788449.14 total 1612.74 last 29.02) - open/close 41784 (time 1709788459.16 total 1622.76 last 29.45) - open/close 42072 (time 1709788469.18 total 1632.79 last 28.73) - open/close 42370 (time 1709788479.19 total 1642.80 last 29.78) - open/close 42644 (time 1709788489.22 total 1652.82 last 27.33) - open/close 42912 (time 1709788499.23 total 1662.84 last 26.76) - open/close 43193 (time 1709788509.25 total 1672.86 last 28.04) - open/close 43487 (time 1709788519.27 total 1682.88 last 29.35) - open/close 43756 (time 1709788529.28 total 1692.89 last 26.87) - open/close 44040 (time 1709788539.31 total 1702.92 last 28.31) - open/close 44318 (time 1709788549.32 total 1712.93 last 27.76) - open/close 44594 (time 1709788559.35 total 1722.95 last 27.54) - open/close 44872 (time 1709788569.37 total 1732.98 last 27.72) - open/close 45156 (time 1709788579.40 total 1743.01 last 28.33) - open/close 45432 (time 1709788589.40 total 1753.01 last 27.60) - open/close 45705 (time 1709788599.43 total 1763.04 last 27.22) - open/close 45980 (time 1709788609.46 total 1773.07 last 27.42) - open/close 46255 (time 1709788619.48 total 1783.08 last 27.46) - open/close 46520 (time 1709788629.48 total 1793.08 last 26.50) - open/close 46773 (time 1709788639.50 total 1803.10 last 25.25) - open/close 46972 (time 1709788649.51 total 1813.11 last 19.88) - open/close 47168 (time 1709788659.53 total 1823.14 last 19.56) - open/close 47371 (time 1709788669.55 total 1833.16 last 20.25) - open/close 47619 (time 1709788679.55 total 1843.16 last 24.80) - open/close 47860 (time 1709788689.57 total 1853.18 last 24.05) - open/close 48114 (time 1709788699.59 total 1863.20 last 25.36) - open/close 48363 (time 1709788709.64 total 1873.25 last 24.78) - open/close 48612 (time 1709788719.68 total 1883.29 last 24.80) - open/close 48867 (time 1709788729.71 total 1893.32 last 25.41) - open/close 49131 (time 1709788739.72 total 1903.32 last 26.39) - open/close 49394 (time 1709788749.74 total 1913.34 last 26.25) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No space left on device total: 49600 open/close in 1921.59 seconds: 25.81 ops/second - unlinked 0 (time 1709788759 ; total 0 ; last 0) - unlinked 10000 (time 1709789205 ; total 446 ; last 446) - unlinked 20000 (time 1709789567 ; total 808 ; last 362) - unlinked 30000 (time 1709789831 ; total 1072 ; last 264) - unlinked 40000 (time 1709789964 ; total 1205 ; last 133) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49600) error: No such file or directory total: 49600 unlinks in 1250 seconds: 39.680000 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg311-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg311-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg311-server stop mds service on oleg311-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg311-server unloading modules on: 'oleg311-server' oleg311-server: oleg311-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-client: ssh exited with exit code 2 pdsh@oleg311-client: oleg311-server: ssh exited with exit code 2