== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:03 (1709786763) start mds service on oleg449-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg449-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg449-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg449-server: oleg449-server.virtnet: executing set_default_debug -1 all pdsh@oleg449-client: oleg449-server: ssh exited with exit code 1 Started lustre-OST0000 oleg449-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg449-client.virtnet: -o user_xattr,flock oleg449-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 479 (time 1709786794.68 total 10.00 last 47.90) - open/close 714 (time 1709786804.68 total 20.00 last 23.49) - open/close 925 (time 1709786814.69 total 30.01 last 21.09) - open/close 1230 (time 1709786824.97 total 40.29 last 29.67) - open/close 1416 (time 1709786835.28 total 50.60 last 18.03) - open/close 1633 (time 1709786845.57 total 60.89 last 21.10) - open/close 1953 (time 1709786855.57 total 70.89 last 31.98) - open/close 2081 (time 1709786866.00 total 81.32 last 12.27) - open/close 2432 (time 1709786876.36 total 91.68 last 33.89) - open/close 2782 (time 1709786886.37 total 101.69 last 34.96) - open/close 2983 (time 1709786896.77 total 112.09 last 19.33) - open/close 3211 (time 1709786906.78 total 122.10 last 22.77) - open/close 3467 (time 1709786917.57 total 132.90 last 23.72) - open/close 3668 (time 1709786927.99 total 143.31 last 19.30) - open/close 3979 (time 1709786938.74 total 154.06 last 28.92) - open/close 4220 (time 1709786948.75 total 164.07 last 24.09) - open/close 4436 (time 1709786958.97 total 174.29 last 21.13) - open/close 4580 (time 1709786969.43 total 184.75 last 13.77) - open/close 4742 (time 1709786979.66 total 194.98 last 15.83) - open/close 4999 (time 1709786990.08 total 205.40 last 24.66) - open/close 5214 (time 1709787000.09 total 215.41 last 21.48) - open/close 5374 (time 1709787010.93 total 226.25 last 14.76) - open/close 5545 (time 1709787021.16 total 236.48 last 16.72) - open/close 5820 (time 1709787031.17 total 246.49 last 27.48) - open/close 5988 (time 1709787041.75 total 257.07 last 15.87) - open/close 6202 (time 1709787052.51 total 267.83 last 19.90) - open/close 6538 (time 1709787063.48 total 278.80 last 30.63) - open/close 6753 (time 1709787074.03 total 289.35 last 20.37) - open/close 6905 (time 1709787084.92 total 300.24 last 13.95) - open/close 7197 (time 1709787095.72 total 311.05 last 27.03) - open/close 7393 (time 1709787105.73 total 321.05 last 19.59) - open/close 7701 (time 1709787116.69 total 332.01 last 28.09) - open/close 7896 (time 1709787127.10 total 342.42 last 18.74) - open/close 7922 (time 1709787137.60 total 352.92 last 2.48) - open/close 8121 (time 1709787149.34 total 364.66 last 16.95) - open/close 8391 (time 1709787161.54 total 376.86 last 22.13) - open/close 8637 (time 1709787171.69 total 387.01 last 24.24) - open/close 8883 (time 1709787182.24 total 397.56 last 23.31) - open/close 9060 (time 1709787192.45 total 407.77 last 17.33) - open/close 9241 (time 1709787203.30 total 418.62 last 16.69) - open/close 9426 (time 1709787213.83 total 429.15 last 17.57) - open/close 9551 (time 1709787224.75 total 440.07 last 11.44) - open/close 9727 (time 1709787235.64 total 450.96 last 16.17) - open/close 9878 (time 1709787245.79 total 461.11 last 14.88) - open/close 9956 (time 1709787256.71 total 472.03 last 7.14) - open/close 10502 (time 1709787266.75 total 482.07 last 54.39) - open/close 11008 (time 1709787276.77 total 492.09 last 50.49) - open/close 11456 (time 1709787286.81 total 502.13 last 44.66) - open/close 11944 (time 1709787296.81 total 512.13 last 48.77) - open/close 12389 (time 1709787306.82 total 522.14 last 44.45) - open/close 12835 (time 1709787316.84 total 532.16 last 44.53) - open/close 13302 (time 1709787326.85 total 542.17 last 46.66) - open/close 13745 (time 1709787336.86 total 552.18 last 44.24) - open/close 14184 (time 1709787346.86 total 562.18 last 43.90) - open/close 14550 (time 1709787366.50 total 581.82 last 18.64) - open/close 14948 (time 1709787376.51 total 591.83 last 39.76) - open/close 15313 (time 1709787386.53 total 601.85 last 36.42) - open/close 15698 (time 1709787396.54 total 611.86 last 38.46) - open/close 16077 (time 1709787406.54 total 621.86 last 37.89) - open/close 16408 (time 1709787416.56 total 631.88 last 33.04) - open/close 16718 (time 1709787426.59 total 641.91 last 30.90) - open/close 17063 (time 1709787436.60 total 651.92 last 34.47) - open/close 17382 (time 1709787446.60 total 661.93 last 31.89) - open/close 17725 (time 1709787456.61 total 671.93 last 34.28) - open/close 18046 (time 1709787466.62 total 681.94 last 32.07) - open/close 18326 (time 1709787476.63 total 691.95 last 27.97) - open/close 18651 (time 1709787486.65 total 701.97 last 32.45) - open/close 18982 (time 1709787496.65 total 711.97 last 33.09) - open/close 19291 (time 1709787506.70 total 722.02 last 30.74) - open/close 19574 (time 1709787516.71 total 732.03 last 28.29) - open/close 19854 (time 1709787526.73 total 742.05 last 27.94) - open/close 20000 (time 1709787531.74 total 747.06 last 29.14) - open/close 20293 (time 1709787541.76 total 757.08 last 29.24) - open/close 20606 (time 1709787551.76 total 767.08 last 31.30) - open/close 20917 (time 1709787561.79 total 777.11 last 31.02) - open/close 21264 (time 1709787571.80 total 787.12 last 34.66) - open/close 21591 (time 1709787581.81 total 797.13 last 32.67) - open/close 21799 (time 1709787598.84 total 814.16 last 12.21) - open/close 22150 (time 1709787608.86 total 824.18 last 35.04) - open/close 22471 (time 1709787618.88 total 834.20 last 32.03) - open/close 22803 (time 1709787628.89 total 844.22 last 33.15) - open/close 23124 (time 1709787638.90 total 854.22 last 32.08) - open/close 23445 (time 1709787648.92 total 864.24 last 32.06) - open/close 23787 (time 1709787658.94 total 874.26 last 34.12) - open/close 24110 (time 1709787668.96 total 884.28 last 32.23) - open/close 24397 (time 1709787678.98 total 894.30 last 28.66) - open/close 24704 (time 1709787689.00 total 904.32 last 30.62) - open/close 25011 (time 1709787699.01 total 914.33 last 30.66) - open/close 25298 (time 1709787709.03 total 924.35 last 28.66) - open/close 25563 (time 1709787719.05 total 934.37 last 26.44) - open/close 25853 (time 1709787729.07 total 944.39 last 28.94) - open/close 26166 (time 1709787739.09 total 954.41 last 31.26) - open/close 26473 (time 1709787749.09 total 964.41 last 30.68) - open/close 26739 (time 1709787759.10 total 974.42 last 26.59) - open/close 26982 (time 1709787769.13 total 984.46 last 24.21) - open/close 27221 (time 1709787779.17 total 994.49 last 23.81) - open/close 27466 (time 1709787789.20 total 1004.52 last 24.43) - open/close 27731 (time 1709787799.22 total 1014.54 last 26.46) - open/close 27860 (time 1709787815.01 total 1030.33 last 8.17) - open/close 28125 (time 1709787825.05 total 1040.37 last 26.39) - open/close 28397 (time 1709787835.07 total 1050.39 last 27.16) - open/close 28693 (time 1709787845.07 total 1060.39 last 29.58) - open/close 29002 (time 1709787855.09 total 1070.41 last 30.84) - open/close 29337 (time 1709787865.10 total 1080.42 last 33.46) - open/close 29646 (time 1709787875.11 total 1090.43 last 30.89) - open/close 29975 (time 1709787885.11 total 1100.43 last 32.88) - open/close 30289 (time 1709787895.13 total 1110.45 last 31.35) - open/close 30591 (time 1709787905.16 total 1120.48 last 30.10) - open/close 30868 (time 1709787915.19 total 1130.51 last 27.62) - open/close 31157 (time 1709787925.23 total 1140.55 last 28.80) - open/close 31465 (time 1709787935.24 total 1150.56 last 30.77) - open/close 31751 (time 1709787945.27 total 1160.59 last 28.51) - open/close 32046 (time 1709787955.29 total 1170.61 last 29.42) - open/close 32340 (time 1709787965.36 total 1180.69 last 29.19) - open/close 32618 (time 1709787975.41 total 1190.73 last 27.69) - open/close 32907 (time 1709787985.42 total 1200.74 last 28.86) - open/close 33210 (time 1709787995.43 total 1210.75 last 30.26) - open/close 33481 (time 1709788005.43 total 1220.75 last 27.10) - open/close 33747 (time 1709788015.45 total 1230.77 last 26.57) - open/close 33996 (time 1709788025.47 total 1240.79 last 24.83) - open/close 34275 (time 1709788035.53 total 1250.85 last 27.76) - open/close 34555 (time 1709788045.54 total 1260.86 last 27.95) - open/close 34795 (time 1709788055.54 total 1270.87 last 24.00) - open/close 35070 (time 1709788065.55 total 1280.87 last 27.50) - open/close 35355 (time 1709788075.56 total 1290.88 last 28.46) - open/close 35614 (time 1709788085.57 total 1300.89 last 25.88) - open/close 35883 (time 1709788095.58 total 1310.90 last 26.88) - open/close 36167 (time 1709788105.60 total 1320.92 last 28.33) - open/close 36425 (time 1709788115.63 total 1330.95 last 25.74) - open/close 36710 (time 1709788125.66 total 1340.98 last 28.40) - open/close 36949 (time 1709788135.68 total 1351.00 last 23.86) - open/close 37194 (time 1709788145.70 total 1361.02 last 24.45) - open/close 37433 (time 1709788155.70 total 1371.02 last 23.90) - open/close 37659 (time 1709788165.72 total 1381.04 last 22.55) - open/close 37888 (time 1709788175.72 total 1391.04 last 22.90) - open/close 38143 (time 1709788185.74 total 1401.06 last 25.45) - open/close 38391 (time 1709788195.78 total 1411.10 last 24.70) - open/close 38620 (time 1709788205.80 total 1421.13 last 22.85) - open/close 38872 (time 1709788215.84 total 1431.16 last 25.12) - open/close 39107 (time 1709788225.86 total 1441.18 last 23.44) - open/close 39358 (time 1709788235.87 total 1451.19 last 25.08) - open/close 39580 (time 1709788245.89 total 1461.21 last 22.14) - open/close 39828 (time 1709788255.92 total 1471.24 last 24.73) - open/close 40000 (time 1709788262.21 total 1477.53 last 27.37) - open/close 40249 (time 1709788272.21 total 1487.53 last 24.89) - open/close 40490 (time 1709788282.25 total 1497.57 last 24.01) - open/close 40741 (time 1709788292.27 total 1507.59 last 25.05) - open/close 40986 (time 1709788302.32 total 1517.64 last 24.37) - open/close 41214 (time 1709788312.35 total 1527.67 last 22.73) - open/close 41435 (time 1709788322.36 total 1537.68 last 22.08) - open/close 41679 (time 1709788332.37 total 1547.69 last 24.36) - open/close 41919 (time 1709788342.40 total 1557.72 last 23.93) - open/close 42149 (time 1709788352.41 total 1567.73 last 22.99) - open/close 42367 (time 1709788362.43 total 1577.75 last 21.76) - open/close 42606 (time 1709788372.43 total 1587.75 last 23.88) - open/close 42842 (time 1709788382.46 total 1597.78 last 23.53) - open/close 43077 (time 1709788392.50 total 1607.82 last 23.41) - open/close 43315 (time 1709788402.53 total 1617.85 last 23.75) - open/close 43550 (time 1709788412.58 total 1627.90 last 23.37) - open/close 43768 (time 1709788422.60 total 1637.92 last 21.76) - open/close 43987 (time 1709788432.64 total 1647.96 last 21.81) - open/close 44192 (time 1709788442.68 total 1658.00 last 20.42) - open/close 44415 (time 1709788452.69 total 1668.01 last 22.28) - open/close 44641 (time 1709788462.72 total 1678.04 last 22.53) - open/close 44875 (time 1709788472.73 total 1688.05 last 23.38) - open/close 45089 (time 1709788482.75 total 1698.07 last 21.35) - open/close 45311 (time 1709788492.75 total 1708.07 last 22.19) - open/close 45538 (time 1709788502.78 total 1718.10 last 22.65) - open/close 45781 (time 1709788512.78 total 1728.10 last 24.29) - open/close 45997 (time 1709788522.79 total 1738.11 last 21.58) - open/close 46205 (time 1709788532.79 total 1748.11 last 20.80) - open/close 46433 (time 1709788542.83 total 1758.15 last 22.70) - open/close 46661 (time 1709788552.87 total 1768.19 last 22.72) - open/close 46871 (time 1709788562.88 total 1778.20 last 20.97) - open/close 47088 (time 1709788572.91 total 1788.23 last 21.65) - open/close 47305 (time 1709788582.94 total 1798.26 last 21.63) - open/close 47527 (time 1709788592.94 total 1808.26 last 22.20) - open/close 47754 (time 1709788602.98 total 1818.30 last 22.60) - open/close 47967 (time 1709788612.99 total 1828.31 last 21.29) - open/close 48196 (time 1709788623.02 total 1838.34 last 22.83) - open/close 48429 (time 1709788633.04 total 1848.36 last 23.27) - open/close 48658 (time 1709788643.07 total 1858.39 last 22.82) - open/close 48875 (time 1709788653.11 total 1868.43 last 21.63) - open/close 49102 (time 1709788663.12 total 1878.44 last 22.66) - open/close 49286 (time 1709788673.17 total 1888.49 last 18.31) - open/close 49445 (time 1709788683.23 total 1898.55 last 15.82) - open/close 49621 (time 1709788693.27 total 1908.59 last 17.52) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1912.68 seconds: 25.98 ops/second - unlinked 0 (time 1709788698 ; total 0 ; last 0) - unlinked 10000 (time 1709789230 ; total 532 ; last 532) - unlinked 20000 (time 1709789658 ; total 960 ; last 428) - unlinked 30000 (time 1709789962 ; total 1264 ; last 304) - unlinked 40000 (time 1709790101 ; total 1403 ; last 139) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1455 seconds: 34.156700 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg449-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg449-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg449-server stop mds service on oleg449-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg449-server unloading modules on: 'oleg449-server' oleg449-server: oleg449-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-client: ssh exited with exit code 2 pdsh@oleg449-client: oleg449-server: ssh exited with exit code 2