== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:51:41 (1709787101) start mds service on oleg147-server Starting mds1: -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg147-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg147-server Starting ost1: -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg147-server: oleg147-server.virtnet: executing set_default_debug -1 all pdsh@oleg147-client: oleg147-server: ssh exited with exit code 1 Started lustre-OST0000 oleg147-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg147-client.virtnet: -o user_xattr,flock oleg147-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 270 (time 1709787135.07 total 10.15 last 26.61) - open/close 388 (time 1709787145.53 total 20.61 last 11.27) - open/close 543 (time 1709787156.66 total 31.74 last 13.94) - open/close 740 (time 1709787167.59 total 42.67 last 18.02) - open/close 900 (time 1709787178.06 total 53.14 last 15.28) - open/close 1026 (time 1709787188.15 total 63.22 last 12.50) - open/close 1176 (time 1709787198.15 total 73.23 last 15.00) - open/close 1442 (time 1709787208.19 total 83.27 last 26.50) - open/close 1498 (time 1709787221.40 total 96.48 last 4.24) - open/close 1507 (time 1709787232.64 total 107.72 last 0.80) - open/close 1653 (time 1709787242.70 total 117.78 last 14.51) - open/close 1788 (time 1709787252.70 total 127.78 last 13.49) - open/close 1923 (time 1709787263.68 total 138.76 last 12.30) - open/close 2063 (time 1709787274.46 total 149.54 last 12.99) - open/close 2301 (time 1709787285.22 total 160.30 last 22.12) - open/close 2519 (time 1709787295.91 total 170.99 last 20.39) - open/close 2710 (time 1709787306.81 total 181.89 last 17.53) - open/close 2915 (time 1709787316.82 total 191.90 last 20.47) - open/close 3086 (time 1709787326.90 total 201.98 last 16.96) - open/close 3245 (time 1709787336.93 total 212.00 last 15.86) - open/close 3436 (time 1709787347.33 total 222.41 last 18.36) - open/close 3619 (time 1709787357.35 total 232.43 last 18.26) - open/close 3826 (time 1709787367.95 total 243.03 last 19.54) - open/close 4011 (time 1709787377.95 total 253.03 last 18.49) - open/close 4201 (time 1709787388.84 total 263.92 last 17.45) - open/close 4469 (time 1709787399.57 total 274.65 last 24.98) - open/close 4628 (time 1709787410.30 total 285.37 last 14.82) - open/close 4862 (time 1709787420.56 total 295.64 last 22.79) - open/close 5118 (time 1709787430.62 total 305.70 last 25.45) - open/close 5239 (time 1709787441.17 total 316.25 last 11.47) - open/close 5438 (time 1709787451.83 total 326.91 last 18.68) - open/close 5577 (time 1709787462.80 total 337.88 last 12.66) - open/close 5756 (time 1709787472.81 total 347.88 last 17.89) - open/close 5970 (time 1709787483.63 total 358.71 last 19.76) - open/close 6168 (time 1709787494.43 total 369.51 last 18.34) - open/close 6312 (time 1709787505.23 total 380.31 last 13.33) - open/close 6639 (time 1709787515.23 total 390.31 last 32.69) - open/close 6855 (time 1709787525.25 total 400.33 last 21.57) - open/close 6986 (time 1709787535.90 total 410.98 last 12.29) - open/close 7120 (time 1709787546.10 total 421.18 last 13.14) - open/close 7290 (time 1709787557.04 total 432.12 last 15.54) - open/close 7488 (time 1709787567.56 total 442.63 last 18.83) - open/close 7681 (time 1709787577.58 total 452.66 last 19.25) - open/close 7811 (time 1709787588.31 total 463.39 last 12.11) - open/close 8054 (time 1709787599.19 total 474.27 last 22.34) - open/close 8254 (time 1709787610.18 total 485.26 last 18.20) - open/close 8401 (time 1709787622.52 total 497.60 last 11.91) - open/close 8647 (time 1709787632.52 total 507.60 last 24.60) - open/close 8871 (time 1709787642.54 total 517.62 last 22.35) - open/close 9003 (time 1709787652.60 total 527.68 last 13.13) - open/close 9160 (time 1709787663.05 total 538.13 last 15.02) - open/close 9288 (time 1709787673.93 total 549.01 last 11.76) - open/close 9466 (time 1709787684.80 total 559.88 last 16.37) - open/close 9705 (time 1709787695.10 total 570.18 last 23.21) - open/close 9893 (time 1709787705.12 total 580.20 last 18.77) - open/close 10000 (time 1709787708.08 total 583.16 last 36.13) - open/close 10653 (time 1709787718.08 total 593.16 last 65.26) - open/close 11301 (time 1709787728.10 total 603.18 last 64.72) - open/close 11883 (time 1709787738.11 total 613.19 last 58.13) - open/close 12404 (time 1709787748.12 total 623.20 last 52.06) - open/close 12922 (time 1709787758.13 total 633.21 last 51.75) - open/close 13422 (time 1709787768.13 total 643.21 last 49.97) - open/close 13888 (time 1709787778.13 total 653.21 last 46.59) - open/close 14335 (time 1709787788.15 total 663.22 last 44.65) - open/close 14755 (time 1709787798.15 total 673.23 last 41.96) - open/close 15209 (time 1709787808.16 total 683.24 last 45.36) - open/close 15689 (time 1709787818.17 total 693.25 last 47.94) - open/close 16140 (time 1709787828.18 total 703.26 last 45.05) - open/close 16522 (time 1709787838.20 total 713.28 last 38.13) - open/close 16910 (time 1709787848.20 total 723.28 last 38.80) - open/close 17313 (time 1709787858.21 total 733.29 last 40.27) - open/close 17714 (time 1709787868.22 total 743.30 last 40.06) - open/close 18106 (time 1709787878.23 total 753.31 last 39.15) - open/close 18478 (time 1709787888.25 total 763.33 last 37.14) - open/close 18847 (time 1709787898.27 total 773.35 last 36.81) - open/close 19191 (time 1709787908.28 total 783.36 last 34.35) - open/close 19552 (time 1709787918.29 total 793.37 last 36.06) - open/close 19870 (time 1709787928.32 total 803.40 last 31.73) - open/close 20000 (time 1709787931.94 total 807.02 last 35.90) - open/close 20337 (time 1709787941.96 total 817.04 last 33.62) - open/close 20702 (time 1709787951.97 total 827.05 last 36.49) - open/close 21116 (time 1709787961.97 total 837.05 last 41.37) - open/close 21484 (time 1709787971.98 total 847.06 last 36.79) - open/close 21908 (time 1709787982.00 total 857.08 last 42.32) - open/close 22332 (time 1709787992.00 total 867.08 last 42.38) - open/close 22735 (time 1709788002.00 total 877.08 last 40.29) - open/close 23034 (time 1709788020.80 total 895.88 last 15.91) - open/close 23388 (time 1709788030.80 total 905.88 last 35.39) - open/close 23769 (time 1709788040.81 total 915.89 last 38.09) - open/close 24167 (time 1709788050.82 total 925.90 last 39.76) - open/close 24539 (time 1709788060.82 total 935.90 last 37.18) - open/close 24905 (time 1709788070.85 total 945.93 last 36.49) - open/close 25309 (time 1709788080.85 total 955.93 last 40.39) - open/close 25700 (time 1709788090.85 total 965.93 last 39.10) - open/close 26054 (time 1709788100.86 total 975.94 last 35.38) - open/close 26397 (time 1709788110.88 total 985.95 last 34.24) - open/close 26766 (time 1709788120.90 total 995.98 last 36.81) - open/close 27086 (time 1709788130.90 total 1005.98 last 32.00) - open/close 27447 (time 1709788140.90 total 1015.98 last 36.10) - open/close 27775 (time 1709788150.91 total 1025.99 last 32.76) - open/close 28098 (time 1709788160.92 total 1036.00 last 32.28) - open/close 28398 (time 1709788170.93 total 1046.01 last 29.97) - open/close 28729 (time 1709788180.93 total 1056.01 last 33.09) - open/close 29053 (time 1709788190.96 total 1066.04 last 32.33) - open/close 29392 (time 1709788200.96 total 1076.04 last 33.88) - open/close 29760 (time 1709788210.97 total 1086.05 last 36.77) - open/close 30000 (time 1709788218.01 total 1093.09 last 34.09) - open/close 30331 (time 1709788228.01 total 1103.09 last 33.10) - open/close 30680 (time 1709788238.03 total 1113.11 last 34.84) - open/close 30991 (time 1709788248.04 total 1123.12 last 31.05) - open/close 31269 (time 1709788258.05 total 1133.13 last 27.77) - open/close 31569 (time 1709788268.08 total 1143.16 last 29.93) - open/close 31874 (time 1709788278.08 total 1153.16 last 30.48) - open/close 32197 (time 1709788288.11 total 1163.19 last 32.20) - open/close 32516 (time 1709788298.13 total 1173.21 last 31.83) - open/close 32828 (time 1709788308.16 total 1183.24 last 31.13) - open/close 33126 (time 1709788318.19 total 1193.27 last 29.71) - open/close 33436 (time 1709788328.19 total 1203.27 last 30.98) - open/close 33739 (time 1709788338.22 total 1213.30 last 30.22) - open/close 34022 (time 1709788348.25 total 1223.33 last 28.21) - open/close 34325 (time 1709788358.28 total 1233.36 last 30.21) - open/close 34622 (time 1709788368.29 total 1243.36 last 29.68) - open/close 34921 (time 1709788378.29 total 1253.36 last 29.90) - open/close 35219 (time 1709788388.31 total 1263.39 last 29.72) - open/close 35513 (time 1709788398.34 total 1273.42 last 29.32) - open/close 35806 (time 1709788408.36 total 1283.44 last 29.25) - open/close 36092 (time 1709788418.39 total 1293.47 last 28.52) - open/close 36375 (time 1709788428.39 total 1303.47 last 28.29) - open/close 36650 (time 1709788438.40 total 1313.48 last 27.47) - open/close 36900 (time 1709788448.43 total 1323.51 last 24.95) - open/close 37168 (time 1709788458.46 total 1333.54 last 26.72) - open/close 37427 (time 1709788468.46 total 1343.54 last 25.90) - open/close 37721 (time 1709788478.49 total 1353.57 last 29.30) - open/close 37996 (time 1709788488.52 total 1363.60 last 27.42) - open/close 38280 (time 1709788498.53 total 1373.61 last 28.37) - open/close 38563 (time 1709788508.57 total 1383.65 last 28.21) - open/close 38837 (time 1709788518.59 total 1393.67 last 27.32) - open/close 39110 (time 1709788528.60 total 1403.68 last 27.27) - open/close 39394 (time 1709788538.64 total 1413.72 last 28.31) - open/close 39677 (time 1709788548.67 total 1423.75 last 28.20) - open/close 39947 (time 1709788558.70 total 1433.78 last 26.94) - open/close 40224 (time 1709788568.72 total 1443.80 last 27.64) - open/close 40495 (time 1709788578.76 total 1453.84 last 26.99) - open/close 40761 (time 1709788588.76 total 1463.84 last 26.60) - open/close 41028 (time 1709788598.78 total 1473.86 last 26.65) - open/close 41290 (time 1709788608.80 total 1483.88 last 26.15) - open/close 41559 (time 1709788618.83 total 1493.91 last 26.80) - open/close 41833 (time 1709788628.87 total 1503.95 last 27.30) - open/close 42086 (time 1709788638.90 total 1513.98 last 25.23) - open/close 42339 (time 1709788648.94 total 1524.02 last 25.20) - open/close 42589 (time 1709788658.98 total 1534.06 last 24.90) - open/close 42838 (time 1709788669.02 total 1544.10 last 24.80) - open/close 43074 (time 1709788679.04 total 1554.12 last 23.55) - open/close 43309 (time 1709788689.06 total 1564.14 last 23.44) - open/close 43555 (time 1709788699.08 total 1574.16 last 24.55) - open/close 43802 (time 1709788709.08 total 1584.16 last 24.70) - open/close 44046 (time 1709788719.12 total 1594.20 last 24.32) - open/close 44286 (time 1709788729.13 total 1604.21 last 23.96) - open/close 44517 (time 1709788739.16 total 1614.24 last 23.05) - open/close 44751 (time 1709788749.18 total 1624.26 last 23.34) - open/close 44990 (time 1709788759.20 total 1634.28 last 23.85) - open/close 45218 (time 1709788769.21 total 1644.29 last 22.79) - open/close 45447 (time 1709788779.22 total 1654.30 last 22.87) - open/close 45685 (time 1709788789.24 total 1664.32 last 23.76) - open/close 45908 (time 1709788799.24 total 1674.32 last 22.30) - open/close 46135 (time 1709788809.27 total 1684.35 last 22.62) - open/close 46364 (time 1709788819.27 total 1694.35 last 22.89) - open/close 46588 (time 1709788829.31 total 1704.38 last 22.33) - open/close 46812 (time 1709788839.34 total 1714.42 last 22.31) - open/close 47039 (time 1709788849.38 total 1724.46 last 22.61) - open/close 47253 (time 1709788859.40 total 1734.48 last 21.36) - open/close 47465 (time 1709788869.43 total 1744.51 last 21.15) - open/close 47680 (time 1709788879.46 total 1754.54 last 21.42) - open/close 47884 (time 1709788889.47 total 1764.55 last 20.39) - open/close 48101 (time 1709788899.50 total 1774.58 last 21.63) - open/close 48311 (time 1709788909.52 total 1784.60 last 20.96) - open/close 48522 (time 1709788919.55 total 1794.63 last 21.03) - open/close 48730 (time 1709788929.57 total 1804.65 last 20.77) - open/close 48945 (time 1709788939.58 total 1814.66 last 21.46) - open/close 49157 (time 1709788949.61 total 1824.69 last 21.15) - open/close 49363 (time 1709788959.61 total 1834.69 last 20.59) - open/close 49569 (time 1709788969.63 total 1844.71 last 20.56) open(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No space left on device total: 49698 open/close in 1850.85 seconds: 26.85 ops/second - unlinked 0 (time 1709788977 ; total 0 ; last 0) - unlinked 10000 (time 1709789536 ; total 559 ; last 559) - unlinked 20000 (time 1709789996 ; total 1019 ; last 460) - unlinked 30000 (time 1709790311 ; total 1334 ; last 315) - unlinked 40000 (time 1709790446 ; total 1469 ; last 135) unlink(/mnt/lustre/d101a.conf-sanity/f101a.conf-sanity-49698) error: No such file or directory total: 49698 unlinks in 1522 seconds: 32.653088 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg147-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg147-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg147-server stop mds service on oleg147-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg147-server unloading modules on: 'oleg147-server' oleg147-server: oleg147-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-client: ssh exited with exit code 2 pdsh@oleg147-client: oleg147-server: ssh exited with exit code 2