== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:46:17 (1709786777) start mds service on oleg305-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg305-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg305-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg305-server: oleg305-server.virtnet: executing set_default_debug -1 all pdsh@oleg305-client: oleg305-server: ssh exited with exit code 1 Started lustre-OST0000 oleg305-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg305-client.virtnet: -o user_xattr,flock oleg305-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 276 (time 1709786807.68 total 10.97 last 25.16) - open/close 640 (time 1709786817.69 total 20.97 last 36.38) - open/close 851 (time 1709786828.11 total 31.39 last 20.25) - open/close 1049 (time 1709786838.18 total 41.46 last 19.66) - open/close 1269 (time 1709786848.67 total 51.95 last 20.97) - open/close 1551 (time 1709786858.68 total 61.96 last 28.18) - open/close 1812 (time 1709786868.68 total 71.96 last 26.10) - open/close 2107 (time 1709786878.69 total 81.97 last 29.48) - open/close 2322 (time 1709786889.47 total 92.75 last 19.94) - open/close 2568 (time 1709786899.86 total 103.15 last 23.67) - open/close 2816 (time 1709786910.56 total 113.84 last 23.19) - open/close 3096 (time 1709786921.51 total 124.79 last 25.57) - open/close 3270 (time 1709786931.61 total 134.89 last 17.23) - open/close 3531 (time 1709786942.56 total 145.84 last 23.84) - open/close 3778 (time 1709786952.57 total 155.85 last 24.68) - open/close 3996 (time 1709786963.06 total 166.34 last 20.78) - open/close 4105 (time 1709786973.62 total 176.90 last 10.32) - open/close 4417 (time 1709786983.64 total 186.92 last 31.14) - open/close 4593 (time 1709786994.32 total 197.60 last 16.47) - open/close 4767 (time 1709787004.72 total 208.00 last 16.74) - open/close 4932 (time 1709787015.58 total 218.86 last 15.19) - open/close 5177 (time 1709787025.94 total 229.22 last 23.66) - open/close 5351 (time 1709787035.94 total 239.22 last 17.39) - open/close 5503 (time 1709787046.11 total 249.39 last 14.95) - open/close 5708 (time 1709787056.49 total 259.78 last 19.74) - open/close 6014 (time 1709787066.82 total 270.10 last 29.63) - open/close 6201 (time 1709787077.77 total 281.05 last 17.07) - open/close 6441 (time 1709787088.13 total 291.41 last 23.17) - open/close 6619 (time 1709787099.11 total 302.39 last 16.21) - open/close 6847 (time 1709787109.63 total 312.91 last 21.67) - open/close 6987 (time 1709787119.80 total 323.08 last 13.76) - open/close 7217 (time 1709787130.82 total 334.11 last 20.86) - open/close 7379 (time 1709787140.83 total 344.11 last 16.19) - open/close 7582 (time 1709787153.67 total 356.96 last 15.80) - open/close 7708 (time 1709787166.06 total 369.34 last 10.18) - open/close 7904 (time 1709787176.23 total 379.52 last 19.27) - open/close 8100 (time 1709787187.04 total 390.33 last 18.13) - open/close 8250 (time 1709787197.13 total 400.42 last 14.86) - open/close 8442 (time 1709787207.90 total 411.18 last 17.84) - open/close 8610 (time 1709787217.90 total 421.18 last 16.80) - open/close 8774 (time 1709787228.46 total 431.74 last 15.53) - open/close 8875 (time 1709787238.46 total 441.74 last 10.10) - open/close 9050 (time 1709787248.53 total 451.82 last 17.37) - open/close 9211 (time 1709787259.30 total 462.59 last 14.95) - open/close 9360 (time 1709787269.99 total 473.28 last 13.94) - open/close 9486 (time 1709787281.57 total 484.86 last 10.88) - open/close 9766 (time 1709787291.58 total 494.86 last 27.98) - open/close 10000 (time 1709787295.02 total 498.31 last 67.95) - open/close 10618 (time 1709787305.03 total 508.31 last 61.78) - open/close 11198 (time 1709787315.03 total 518.32 last 57.96) - open/close 11681 (time 1709787325.06 total 528.35 last 48.16) - open/close 12202 (time 1709787335.06 total 538.35 last 52.08) - open/close 12810 (time 1709787345.07 total 548.35 last 60.79) - open/close 13291 (time 1709787355.08 total 558.36 last 48.05) - open/close 13773 (time 1709787365.09 total 568.37 last 48.16) - open/close 14248 (time 1709787375.09 total 578.37 last 47.48) - open/close 14725 (time 1709787385.10 total 588.38 last 47.67) - open/close 15147 (time 1709787395.10 total 598.38 last 42.19) - open/close 15597 (time 1709787405.12 total 608.40 last 44.90) - open/close 16036 (time 1709787415.19 total 618.47 last 43.60) - open/close 16513 (time 1709787425.20 total 628.48 last 47.68) - open/close 16953 (time 1709787435.21 total 638.49 last 43.94) - open/close 17402 (time 1709787445.23 total 648.51 last 44.80) - open/close 17819 (time 1709787455.25 total 658.53 last 41.64) - open/close 18202 (time 1709787465.25 total 668.54 last 38.27) - open/close 18563 (time 1709787475.26 total 678.55 last 36.07) - open/close 18937 (time 1709787485.28 total 688.56 last 37.35) - open/close 19360 (time 1709787495.30 total 698.58 last 42.21) - open/close 19716 (time 1709787505.33 total 708.61 last 35.50) - open/close 20000 (time 1709787513.43 total 716.72 last 35.04) - open/close 20351 (time 1709787523.43 total 726.72 last 35.09) - open/close 20672 (time 1709787533.44 total 736.73 last 32.08) - open/close 21018 (time 1709787543.51 total 746.79 last 34.38) - open/close 21368 (time 1709787553.54 total 756.82 last 34.88) - open/close 21691 (time 1709787563.54 total 766.83 last 32.30) - open/close 22017 (time 1709787573.58 total 776.86 last 32.49) - open/close 22334 (time 1709787583.59 total 786.88 last 31.65) - open/close 22629 (time 1709787593.62 total 796.91 last 29.40) - open/close 22940 (time 1709787603.64 total 806.93 last 31.04) - open/close 23273 (time 1709787613.67 total 816.95 last 33.22) - open/close 23581 (time 1709787623.68 total 826.96 last 30.78) - open/close 23926 (time 1709787633.70 total 836.98 last 34.42) - open/close 24228 (time 1709787643.71 total 847.00 last 30.16) - open/close 24524 (time 1709787653.72 total 857.01 last 29.57) - open/close 24825 (time 1709787663.76 total 867.04 last 29.99) - open/close 25112 (time 1709787673.76 total 877.05 last 28.69) - open/close 25392 (time 1709787683.77 total 887.06 last 27.98) - open/close 25649 (time 1709787693.86 total 897.14 last 25.48) - open/close 25899 (time 1709787703.87 total 907.16 last 24.96) - open/close 26154 (time 1709787713.92 total 917.20 last 25.39) - open/close 26454 (time 1709787723.92 total 927.21 last 29.98) - open/close 26749 (time 1709787733.93 total 937.21 last 29.50) - open/close 27028 (time 1709787743.93 total 947.22 last 27.88) - open/close 27268 (time 1709787753.94 total 957.23 last 23.98) - open/close 27542 (time 1709787763.96 total 967.24 last 27.35) - open/close 27825 (time 1709787773.98 total 977.26 last 28.24) - open/close 28087 (time 1709787784.01 total 987.29 last 26.13) - open/close 28363 (time 1709787794.04 total 997.33 last 27.49) - open/close 28646 (time 1709787814.02 total 1017.30 last 14.17) - open/close 28940 (time 1709787824.04 total 1027.32 last 29.34) - open/close 29232 (time 1709787834.04 total 1037.33 last 29.18) - open/close 29521 (time 1709787844.04 total 1047.33 last 28.89) - open/close 29811 (time 1709787854.06 total 1057.34 last 28.97) - open/close 30000 (time 1709787860.58 total 1063.86 last 28.98) - open/close 30280 (time 1709787870.59 total 1073.87 last 27.96) - open/close 30559 (time 1709787880.60 total 1083.89 last 27.87) - open/close 30813 (time 1709787890.63 total 1093.91 last 25.33) - open/close 31100 (time 1709787900.66 total 1103.95 last 28.60) - open/close 31390 (time 1709787910.69 total 1113.97 last 28.93) - open/close 31675 (time 1709787920.72 total 1124.00 last 28.42) - open/close 31938 (time 1709787930.72 total 1134.00 last 26.30) - open/close 32204 (time 1709787940.76 total 1144.04 last 26.49) - open/close 32348 (time 1709787957.88 total 1161.16 last 8.41) - open/close 32614 (time 1709787967.90 total 1171.18 last 26.55) - open/close 32892 (time 1709787977.91 total 1181.19 last 27.77) - open/close 33187 (time 1709787987.91 total 1191.20 last 29.49) - open/close 33496 (time 1709787997.94 total 1201.22 last 30.83) - open/close 33778 (time 1709788007.96 total 1211.25 last 28.12) - open/close 34043 (time 1709788017.99 total 1221.27 last 26.43) - open/close 34301 (time 1709788028.02 total 1231.31 last 25.72) - open/close 34574 (time 1709788038.04 total 1241.33 last 27.25) - open/close 34819 (time 1709788048.06 total 1251.35 last 24.45) - open/close 35072 (time 1709788058.10 total 1261.38 last 25.21) - open/close 35329 (time 1709788068.14 total 1271.42 last 25.60) - open/close 35632 (time 1709788078.16 total 1281.44 last 30.24) - open/close 35972 (time 1709788088.17 total 1291.45 last 33.95) - open/close 36267 (time 1709788098.19 total 1301.48 last 29.44) - open/close 36563 (time 1709788108.20 total 1311.48 last 29.58) - open/close 36859 (time 1709788118.22 total 1321.50 last 29.54) - open/close 37151 (time 1709788128.24 total 1331.53 last 29.13) - open/close 37451 (time 1709788138.25 total 1341.53 last 29.99) - open/close 37745 (time 1709788148.28 total 1351.56 last 29.30) - open/close 38032 (time 1709788158.30 total 1361.59 last 28.64) - open/close 38320 (time 1709788168.34 total 1371.62 last 28.69) - open/close 38601 (time 1709788178.39 total 1381.67 last 27.96) - open/close 38889 (time 1709788188.42 total 1391.71 last 28.70) - open/close 39165 (time 1709788198.43 total 1401.71 last 27.58) - open/close 39433 (time 1709788208.45 total 1411.74 last 26.73) - open/close 39714 (time 1709788218.47 total 1421.75 last 28.07) - open/close 39999 (time 1709788228.48 total 1431.77 last 28.46) - open/close 40298 (time 1709788238.51 total 1441.79 last 29.82) - open/close 40582 (time 1709788248.53 total 1451.81 last 28.34) - open/close 40847 (time 1709788258.54 total 1461.83 last 26.46) - open/close 41128 (time 1709788268.56 total 1471.84 last 28.06) - open/close 41409 (time 1709788278.58 total 1481.86 last 28.04) - open/close 41680 (time 1709788288.62 total 1491.90 last 27.00) - open/close 41947 (time 1709788298.63 total 1501.91 last 26.67) - open/close 42217 (time 1709788308.64 total 1511.93 last 26.95) - open/close 42490 (time 1709788318.65 total 1521.94 last 27.27) - open/close 42777 (time 1709788328.66 total 1531.94 last 28.70) - open/close 43058 (time 1709788338.68 total 1541.97 last 28.03) - open/close 43341 (time 1709788348.71 total 1552.00 last 28.21) - open/close 43625 (time 1709788358.74 total 1562.03 last 28.31) - open/close 43875 (time 1709788368.75 total 1572.03 last 24.99) - open/close 44141 (time 1709788378.78 total 1582.07 last 26.51) - open/close 44401 (time 1709788388.80 total 1592.08 last 25.97) - open/close 44669 (time 1709788398.82 total 1602.11 last 26.73) - open/close 44925 (time 1709788408.84 total 1612.12 last 25.55) - open/close 45200 (time 1709788418.84 total 1622.13 last 27.49) - open/close 45462 (time 1709788428.89 total 1632.18 last 26.07) - open/close 45710 (time 1709788438.93 total 1642.21 last 24.72) - open/close 45960 (time 1709788448.93 total 1652.21 last 25.00) - open/close 46219 (time 1709788458.96 total 1662.25 last 25.81) - open/close 46465 (time 1709788468.98 total 1672.26 last 24.56) - open/close 46726 (time 1709788478.99 total 1682.27 last 26.08) - open/close 46974 (time 1709788489.01 total 1692.29 last 24.76) - open/close 47216 (time 1709788499.03 total 1702.32 last 24.13) - open/close 47468 (time 1709788509.06 total 1712.34 last 25.14) - open/close 47714 (time 1709788519.08 total 1722.36 last 24.56) - open/close 47958 (time 1709788529.12 total 1732.40 last 24.30) - open/close 48207 (time 1709788539.15 total 1742.43 last 24.83) - open/close 48456 (time 1709788549.17 total 1752.46 last 24.84) - open/close 48701 (time 1709788559.17 total 1762.46 last 24.50) - open/close 48952 (time 1709788569.19 total 1772.48 last 25.05) - open/close 49207 (time 1709788579.22 total 1782.50 last 25.44) - open/close 49459 (time 1709788589.25 total 1792.54 last 25.12) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1801.90 seconds: 27.58 ops/second - unlinked 0 (time 1709788599 ; total 0 ; last 0) - unlinked 10000 (time 1709789057 ; total 458 ; last 458) - unlinked 20000 (time 1709789471 ; total 872 ; last 414) - unlinked 30000 (time 1709789800 ; total 1201 ; last 329) - unlinked 40000 (time 1709789999 ; total 1400 ; last 199) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1464 seconds: 33.946720 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg305-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg305-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg305-server stop mds service on oleg305-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg305-server unloading modules on: 'oleg305-server' oleg305-server: oleg305-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-client: ssh exited with exit code 2 pdsh@oleg305-client: oleg305-server: ssh exited with exit code 2