== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:57:20 (1709787440) start mds service on oleg303-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg303-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg303-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg303-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg303-server: oleg303-server.virtnet: executing set_default_debug -1 all pdsh@oleg303-client: oleg303-server: ssh exited with exit code 1 Started lustre-OST0000 oleg303-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg303-client.virtnet: -o user_xattr,flock oleg303-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 325 (time 1709787485.09 total 10.45 last 31.09) - open/close 465 (time 1709787495.38 total 20.74 last 13.61) - open/close 615 (time 1709787505.99 total 31.35 last 14.13) - open/close 821 (time 1709787516.01 total 41.37 last 20.56) - open/close 1049 (time 1709787526.88 total 52.24 last 20.97) - open/close 1210 (time 1709787537.65 total 63.01 last 14.96) - open/close 1332 (time 1709787548.67 total 74.03 last 11.07) - open/close 1532 (time 1709787559.24 total 84.60 last 18.93) - open/close 1719 (time 1709787569.87 total 95.24 last 17.58) - open/close 1990 (time 1709787580.34 total 105.70 last 25.89) - open/close 2153 (time 1709787590.36 total 115.72 last 16.28) - open/close 2280 (time 1709787600.36 total 125.72 last 12.69) - open/close 2461 (time 1709787610.37 total 135.73 last 18.09) - open/close 2661 (time 1709787621.14 total 146.51 last 18.56) - open/close 2792 (time 1709787631.33 total 156.69 last 12.86) - open/close 2960 (time 1709787641.50 total 166.86 last 16.51) - open/close 3114 (time 1709787652.28 total 177.64 last 14.29) - open/close 3294 (time 1709787663.26 total 188.62 last 16.39) - open/close 3608 (time 1709787673.96 total 199.32 last 29.35) - open/close 3806 (time 1709787684.34 total 209.70 last 19.08) - open/close 4035 (time 1709787694.99 total 220.35 last 21.50) - open/close 4225 (time 1709787705.18 total 230.54 last 18.64) - open/close 4394 (time 1709787715.44 total 240.80 last 16.47) - open/close 4579 (time 1709787725.89 total 251.25 last 17.71) - open/close 4709 (time 1709787736.72 total 262.08 last 12.00) - open/close 4846 (time 1709787746.73 total 272.09 last 13.69) - open/close 5093 (time 1709787756.80 total 282.16 last 24.54) - open/close 5272 (time 1709787767.00 total 292.36 last 17.55) - open/close 5427 (time 1709787777.01 total 302.37 last 15.48) - open/close 5573 (time 1709787787.03 total 312.39 last 14.58) - open/close 5787 (time 1709787797.19 total 322.55 last 21.06) - open/close 5924 (time 1709787807.63 total 332.99 last 13.12) - open/close 6095 (time 1709787817.65 total 343.01 last 17.06) - open/close 6170 (time 1709787828.40 total 353.76 last 6.97) - open/close 6372 (time 1709787838.42 total 363.78 last 20.17) - open/close 6555 (time 1709787849.07 total 374.43 last 17.18) - open/close 6715 (time 1709787859.08 total 384.44 last 15.99) - open/close 6956 (time 1709787870.00 total 395.36 last 22.06) - open/close 7137 (time 1709787880.88 total 406.25 last 16.63) - open/close 7344 (time 1709787890.94 total 416.30 last 20.59) - open/close 7497 (time 1709787900.96 total 426.32 last 15.26) - open/close 7671 (time 1709787910.96 total 436.32 last 17.40) - open/close 7816 (time 1709787921.64 total 447.00 last 13.58) - open/close 7992 (time 1709787932.36 total 457.72 last 16.42) - open/close 8112 (time 1709787942.89 total 468.25 last 11.39) - open/close 8323 (time 1709787953.67 total 479.03 last 19.58) - open/close 8476 (time 1709787963.83 total 489.20 last 15.05) - open/close 8687 (time 1709787973.84 total 499.20 last 21.09) - open/close 8894 (time 1709787983.94 total 509.30 last 20.49) - open/close 9051 (time 1709787994.72 total 520.08 last 14.56) - open/close 9231 (time 1709788005.61 total 530.97 last 16.54) - open/close 9394 (time 1709788017.81 total 543.17 last 13.36) - open/close 9574 (time 1709788028.25 total 553.61 last 17.23) - open/close 9810 (time 1709788039.02 total 564.38 last 21.92) - open/close 10000 (time 1709788048.97 total 574.33 last 19.10) - open/close 10129 (time 1709788065.85 total 591.21 last 7.64) - open/close 10743 (time 1709788075.85 total 601.21 last 61.36) - open/close 11481 (time 1709788085.86 total 611.22 last 73.77) - open/close 12191 (time 1709788095.86 total 621.22 last 70.95) - open/close 12885 (time 1709788105.87 total 631.23 last 69.38) - open/close 13532 (time 1709788115.87 total 641.23 last 64.69) - open/close 14146 (time 1709788125.88 total 651.24 last 61.33) - open/close 14789 (time 1709788135.89 total 661.25 last 64.22) - open/close 15407 (time 1709788145.90 total 671.26 last 61.72) - open/close 15972 (time 1709788155.92 total 681.28 last 56.41) - open/close 16557 (time 1709788165.93 total 691.29 last 58.44) - open/close 17109 (time 1709788175.95 total 701.31 last 55.11) - open/close 17672 (time 1709788185.95 total 711.31 last 56.26) - open/close 18209 (time 1709788195.96 total 721.32 last 53.68) - open/close 18718 (time 1709788205.96 total 731.32 last 50.90) - open/close 19232 (time 1709788215.98 total 741.34 last 51.30) - open/close 19772 (time 1709788225.99 total 751.35 last 53.92) - open/close 20000 (time 1709788230.32 total 755.68 last 52.63) - open/close 20480 (time 1709788240.33 total 765.69 last 47.98) - open/close 20943 (time 1709788250.33 total 775.69 last 46.28) - open/close 21053 (time 1709788263.95 total 789.31 last 8.08) - open/close 21502 (time 1709788273.95 total 799.31 last 44.90) - open/close 21952 (time 1709788283.97 total 809.33 last 44.91) - open/close 22377 (time 1709788293.99 total 819.35 last 42.42) - open/close 22817 (time 1709788304.00 total 829.37 last 43.94) - open/close 23235 (time 1709788314.01 total 839.37 last 41.78) - open/close 23667 (time 1709788324.02 total 849.38 last 43.16) - open/close 24113 (time 1709788334.03 total 859.39 last 44.56) - open/close 24544 (time 1709788344.04 total 869.40 last 43.05) - open/close 24979 (time 1709788354.05 total 879.41 last 43.45) - open/close 25385 (time 1709788364.07 total 889.43 last 40.54) - open/close 25781 (time 1709788374.08 total 899.44 last 39.56) - open/close 26174 (time 1709788384.08 total 909.45 last 39.27) - open/close 26566 (time 1709788394.09 total 919.45 last 39.18) - open/close 26963 (time 1709788404.11 total 929.47 last 39.62) - open/close 27340 (time 1709788414.15 total 939.51 last 37.56) - open/close 27758 (time 1709788424.16 total 949.52 last 41.74) - open/close 28131 (time 1709788434.18 total 959.54 last 37.23) - open/close 28498 (time 1709788444.19 total 969.55 last 36.67) - open/close 28858 (time 1709788454.20 total 979.56 last 35.96) - open/close 29222 (time 1709788464.20 total 989.57 last 36.39) - open/close 29583 (time 1709788474.21 total 999.57 last 36.07) - open/close 29952 (time 1709788484.22 total 1009.58 last 36.89) - open/close 30299 (time 1709788494.22 total 1019.58 last 34.69) - open/close 30643 (time 1709788504.22 total 1029.59 last 34.39) - open/close 30995 (time 1709788514.24 total 1039.60 last 35.16) - open/close 31342 (time 1709788524.24 total 1049.60 last 34.69) - open/close 31686 (time 1709788534.25 total 1059.61 last 34.38) - open/close 32027 (time 1709788544.26 total 1069.62 last 34.05) - open/close 32374 (time 1709788554.26 total 1079.63 last 34.68) - open/close 32715 (time 1709788564.28 total 1089.64 last 34.06) - open/close 33061 (time 1709788574.28 total 1099.64 last 34.60) - open/close 33409 (time 1709788584.30 total 1109.66 last 34.73) - open/close 33758 (time 1709788594.32 total 1119.68 last 34.83) - open/close 34104 (time 1709788604.34 total 1129.70 last 34.53) - open/close 34449 (time 1709788614.36 total 1139.72 last 34.42) - open/close 34802 (time 1709788624.36 total 1149.72 last 35.29) - open/close 35140 (time 1709788634.39 total 1159.75 last 33.72) - open/close 35485 (time 1709788644.41 total 1169.77 last 34.42) - open/close 35822 (time 1709788654.42 total 1179.78 last 33.67) - open/close 36153 (time 1709788664.46 total 1189.82 last 32.97) - open/close 36475 (time 1709788674.48 total 1199.84 last 32.13) - open/close 36787 (time 1709788684.48 total 1209.84 last 31.19) - open/close 37097 (time 1709788694.51 total 1219.87 last 30.91) - open/close 37400 (time 1709788704.52 total 1229.88 last 30.27) - open/close 37708 (time 1709788714.53 total 1239.89 last 30.78) - open/close 38014 (time 1709788724.56 total 1249.93 last 30.50) - open/close 38334 (time 1709788734.58 total 1259.95 last 31.94) - open/close 38654 (time 1709788744.60 total 1269.96 last 31.95) - open/close 38959 (time 1709788754.63 total 1279.99 last 30.40) - open/close 39247 (time 1709788764.65 total 1290.02 last 28.74) - open/close 39526 (time 1709788774.68 total 1300.04 last 27.82) - open/close 39807 (time 1709788784.71 total 1310.07 last 28.02) - open/close 40000 (time 1709788791.78 total 1317.14 last 27.31) - open/close 40257 (time 1709788801.81 total 1327.17 last 25.62) - open/close 40530 (time 1709788811.81 total 1337.17 last 27.29) - open/close 40794 (time 1709788821.82 total 1347.18 last 26.37) - open/close 41061 (time 1709788831.84 total 1357.20 last 26.66) - open/close 41339 (time 1709788841.84 total 1367.21 last 27.78) - open/close 41603 (time 1709788851.85 total 1377.21 last 26.39) - open/close 41873 (time 1709788861.89 total 1387.25 last 26.89) - open/close 42139 (time 1709788871.89 total 1397.25 last 26.59) - open/close 42402 (time 1709788881.91 total 1407.28 last 26.24) - open/close 42682 (time 1709788891.94 total 1417.31 last 27.91) - open/close 42965 (time 1709788901.97 total 1427.33 last 28.22) - open/close 43236 (time 1709788912.00 total 1437.36 last 27.02) - open/close 43514 (time 1709788922.02 total 1447.38 last 27.76) - open/close 43784 (time 1709788932.02 total 1457.38 last 26.98) - open/close 44035 (time 1709788942.06 total 1467.42 last 25.01) - open/close 44290 (time 1709788952.08 total 1477.44 last 25.46) - open/close 44538 (time 1709788962.09 total 1487.45 last 24.76) - open/close 44792 (time 1709788972.11 total 1497.47 last 25.36) - open/close 45051 (time 1709788982.11 total 1507.48 last 25.88) - open/close 45297 (time 1709788992.15 total 1517.51 last 24.51) - open/close 45541 (time 1709789002.17 total 1527.53 last 24.37) - open/close 45790 (time 1709789012.17 total 1537.53 last 24.89) - open/close 46033 (time 1709789022.21 total 1547.57 last 24.21) - open/close 46270 (time 1709789032.22 total 1557.58 last 23.68) - open/close 46497 (time 1709789042.24 total 1567.60 last 22.65) - open/close 46737 (time 1709789052.28 total 1577.64 last 23.91) - open/close 46977 (time 1709789062.28 total 1587.64 last 24.00) - open/close 47213 (time 1709789072.30 total 1597.66 last 23.55) - open/close 47440 (time 1709789082.33 total 1607.69 last 22.63) - open/close 47663 (time 1709789092.34 total 1617.70 last 22.28) - open/close 47894 (time 1709789102.36 total 1627.72 last 23.05) - open/close 48130 (time 1709789112.38 total 1637.74 last 23.55) - open/close 48359 (time 1709789122.41 total 1647.77 last 22.84) - open/close 48581 (time 1709789132.42 total 1657.78 last 22.17) - open/close 48805 (time 1709789142.43 total 1667.80 last 22.36) - open/close 49027 (time 1709789152.44 total 1677.80 last 22.19) - open/close 49255 (time 1709789162.47 total 1687.83 last 22.73) - open/close 49488 (time 1709789172.49 total 1697.86 last 23.24) - open/close 49721 (time 1709789182.50 total 1707.86 last 23.29) - open/close 49944 (time 1709789192.53 total 1717.89 last 22.24) total: 50000 open/close in 1720.43 seconds: 29.06 ops/second - unlinked 0 (time 1709789196 ; total 0 ; last 0) - unlinked 10000 (time 1709789715 ; total 519 ; last 519) - unlinked 20000 (time 1709790145 ; total 949 ; last 430) - unlinked 30000 (time 1709790384 ; total 1188 ; last 239) - unlinked 40000 (time 1709790448 ; total 1252 ; last 64) total: 50000 unlinks in 1291 seconds: 38.729668 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg303-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg303-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg303-server stop mds service on oleg303-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg303-server unloading modules on: 'oleg303-server' oleg303-server: oleg303-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-client: ssh exited with exit code 2 pdsh@oleg303-client: oleg303-server: ssh exited with exit code 2