== conf-sanity test 101a: Race MDT->OST reconnection with create ========================================================== 23:55:05 (1709787305) start mds service on oleg415-server Starting mds1: -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg415-server Starting mds2: -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg415-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg415-server Starting ost1: -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg415-server: oleg415-server.virtnet: executing set_default_debug -1 all pdsh@oleg415-client: oleg415-server: ssh exited with exit code 1 Started lustre-OST0000 oleg415-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg415-client.virtnet: -o user_xattr,flock oleg415-server@tcp:/lustre /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff - open/close 149 (time 1709787351.78 total 10.86 last 13.72) - open/close 346 (time 1709787362.42 total 21.49 last 18.52) - open/close 524 (time 1709787372.94 total 32.01 last 16.92) - open/close 747 (time 1709787382.97 total 42.05 last 22.22) - open/close 1080 (time 1709787393.49 total 52.56 last 31.67) - open/close 1192 (time 1709787404.14 total 63.22 last 10.51) - open/close 1359 (time 1709787414.88 total 73.96 last 15.55) - open/close 1564 (time 1709787425.47 total 84.55 last 19.36) - open/close 1736 (time 1709787435.65 total 94.73 last 16.90) - open/close 1904 (time 1709787446.50 total 105.57 last 15.49) - open/close 2022 (time 1709787456.51 total 115.58 last 11.79) - open/close 2195 (time 1709787466.51 total 125.59 last 17.29) - open/close 2380 (time 1709787479.53 total 138.61 last 14.21) - open/close 2505 (time 1709787489.54 total 148.62 last 12.49) - open/close 2699 (time 1709787499.99 total 159.07 last 18.56) - open/close 2814 (time 1709787510.06 total 169.14 last 11.42) - open/close 2954 (time 1709787520.66 total 179.74 last 13.21) - open/close 3205 (time 1709787530.67 total 189.74 last 25.08) - open/close 3389 (time 1709787541.06 total 200.14 last 17.71) - open/close 3555 (time 1709787551.73 total 210.81 last 15.55) - open/close 3685 (time 1709787561.74 total 220.81 last 12.99) - open/close 4004 (time 1709787571.74 total 230.82 last 31.90) - open/close 4275 (time 1709787581.75 total 240.83 last 27.06) - open/close 4548 (time 1709787591.91 total 250.99 last 26.88) - open/close 4735 (time 1709787602.65 total 261.72 last 17.42) - open/close 4981 (time 1709787613.33 total 272.41 last 23.01) - open/close 5260 (time 1709787623.35 total 282.42 last 27.86) - open/close 5392 (time 1709787633.37 total 292.44 last 13.17) - open/close 5651 (time 1709787643.38 total 302.45 last 25.88) - open/close 5878 (time 1709787653.79 total 312.87 last 21.79) - open/close 6068 (time 1709787664.23 total 323.31 last 18.20) - open/close 6302 (time 1709787674.40 total 333.48 last 23.00) - open/close 6408 (time 1709787684.41 total 343.49 last 10.59) - open/close 6681 (time 1709787695.10 total 354.18 last 25.54) - open/close 6877 (time 1709787705.12 total 364.19 last 19.57) - open/close 7091 (time 1709787716.98 total 376.05 last 18.05) - open/close 7379 (time 1709787727.90 total 386.97 last 26.37) - open/close 7571 (time 1709787738.69 total 397.77 last 17.78) - open/close 7796 (time 1709787748.90 total 407.98 last 22.04) - open/close 7915 (time 1709787758.91 total 417.98 last 11.90) - open/close 8112 (time 1709787768.91 total 427.98 last 19.70) - open/close 8310 (time 1709787779.02 total 438.10 last 19.57) - open/close 8502 (time 1709787790.01 total 449.09 last 17.47) - open/close 8633 (time 1709787800.88 total 459.96 last 12.06) - open/close 8945 (time 1709787811.79 total 470.87 last 28.60) - open/close 9130 (time 1709787822.15 total 481.22 last 17.87) - open/close 9296 (time 1709787832.47 total 491.54 last 16.08) - open/close 9483 (time 1709787843.34 total 502.42 last 17.20) - open/close 10000 (time 1709787851.41 total 510.49 last 64.07) - open/close 10750 (time 1709787861.41 total 520.49 last 74.98) - open/close 11481 (time 1709787871.43 total 530.50 last 72.99) - open/close 12240 (time 1709787881.44 total 540.51 last 75.82) - open/close 12918 (time 1709787891.44 total 550.51 last 67.80) - open/close 13600 (time 1709787901.50 total 560.58 last 67.76) - open/close 14201 (time 1709787911.52 total 570.59 last 60.02) - open/close 14811 (time 1709787921.53 total 580.60 last 60.93) - open/close 15372 (time 1709787931.53 total 590.61 last 56.09) - open/close 15975 (time 1709787941.53 total 600.61 last 60.28) - open/close 16553 (time 1709787951.53 total 610.61 last 57.79) - open/close 17173 (time 1709787961.55 total 620.62 last 61.93) - open/close 17705 (time 1709787971.56 total 630.63 last 53.15) - open/close 18235 (time 1709787981.57 total 640.64 last 52.94) - open/close 18807 (time 1709787991.57 total 650.64 last 57.19) - open/close 19329 (time 1709788001.58 total 660.65 last 52.15) - open/close 19727 (time 1709788011.58 total 670.65 last 39.80) - open/close 20000 (time 1709788018.70 total 677.77 last 38.35) - open/close 20465 (time 1709788028.71 total 687.78 last 46.45) - open/close 20973 (time 1709788038.72 total 697.79 last 50.76) - open/close 21473 (time 1709788048.74 total 707.82 last 49.88) - open/close 21909 (time 1709788058.75 total 717.82 last 43.58) - open/close 22387 (time 1709788068.75 total 727.83 last 47.77) - open/close 22895 (time 1709788078.76 total 737.84 last 50.75) - open/close 23360 (time 1709788088.77 total 747.85 last 46.44) - open/close 23828 (time 1709788098.78 total 757.86 last 46.75) - open/close 24264 (time 1709788108.80 total 767.88 last 43.53) - open/close 24695 (time 1709788118.81 total 777.89 last 43.04) - open/close 25136 (time 1709788128.83 total 787.90 last 44.04) - open/close 25501 (time 1709788138.86 total 797.93 last 36.39) - open/close 25881 (time 1709788148.88 total 807.95 last 37.93) - open/close 26209 (time 1709788158.92 total 817.99 last 32.67) - open/close 26548 (time 1709788168.94 total 828.01 last 33.83) - open/close 26926 (time 1709788178.94 total 838.02 last 37.79) - open/close 27320 (time 1709788188.96 total 848.03 last 39.33) - open/close 27655 (time 1709788198.96 total 858.04 last 33.49) - open/close 28018 (time 1709788208.97 total 868.05 last 36.27) - open/close 28401 (time 1709788219.00 total 878.08 last 38.17) - open/close 28771 (time 1709788229.03 total 888.10 last 36.91) - open/close 29137 (time 1709788239.07 total 898.14 last 36.46) - open/close 29513 (time 1709788249.07 total 908.14 last 37.60) - open/close 29919 (time 1709788259.08 total 918.16 last 40.54) - open/close 30000 (time 1709788261.13 total 920.20 last 39.63) - open/close 30391 (time 1709788271.15 total 930.22 last 39.02) - open/close 30741 (time 1709788281.16 total 940.24 last 34.94) - open/close 31107 (time 1709788291.19 total 950.26 last 36.51) - open/close 31486 (time 1709788301.20 total 960.28 last 37.84) - open/close 31839 (time 1709788311.23 total 970.31 last 35.20) - open/close 32179 (time 1709788321.27 total 980.34 last 33.89) - open/close 32568 (time 1709788331.29 total 990.37 last 38.80) - open/close 32938 (time 1709788341.31 total 1000.39 last 36.91) - open/close 33280 (time 1709788351.35 total 1010.42 last 34.08) - open/close 33611 (time 1709788361.37 total 1020.44 last 33.03) - open/close 33991 (time 1709788371.39 total 1030.46 last 37.93) - open/close 34351 (time 1709788381.40 total 1040.48 last 35.94) - open/close 34711 (time 1709788391.41 total 1050.48 last 35.99) - open/close 35077 (time 1709788401.41 total 1060.49 last 36.57) - open/close 35407 (time 1709788411.43 total 1070.51 last 32.94) - open/close 35713 (time 1709788421.43 total 1080.51 last 30.60) - open/close 36016 (time 1709788431.44 total 1090.52 last 30.28) - open/close 36309 (time 1709788441.47 total 1100.55 last 29.21) - open/close 36604 (time 1709788451.50 total 1110.57 last 29.42) - open/close 36912 (time 1709788461.52 total 1120.60 last 30.73) - open/close 37235 (time 1709788471.53 total 1130.61 last 32.27) - open/close 37551 (time 1709788481.55 total 1140.62 last 31.56) - open/close 37835 (time 1709788491.59 total 1150.66 last 28.29) - open/close 38124 (time 1709788501.62 total 1160.69 last 28.81) - open/close 38462 (time 1709788511.63 total 1170.70 last 33.76) - open/close 38768 (time 1709788521.67 total 1180.75 last 30.47) - open/close 39023 (time 1709788531.71 total 1190.79 last 25.40) - open/close 39309 (time 1709788541.74 total 1200.81 last 28.53) - open/close 39610 (time 1709788551.76 total 1210.84 last 30.02) - open/close 39879 (time 1709788561.79 total 1220.87 last 26.82) - open/close 40000 (time 1709788566.08 total 1225.15 last 28.24) - open/close 40290 (time 1709788576.09 total 1235.16 last 28.97) - open/close 40564 (time 1709788586.10 total 1245.18 last 27.37) - open/close 40867 (time 1709788596.11 total 1255.18 last 30.27) - open/close 41163 (time 1709788606.13 total 1265.21 last 29.53) - open/close 41447 (time 1709788616.14 total 1275.21 last 28.39) - open/close 41737 (time 1709788626.16 total 1285.23 last 28.95) - open/close 42029 (time 1709788636.19 total 1295.26 last 29.11) - open/close 42313 (time 1709788646.21 total 1305.28 last 28.34) - open/close 42583 (time 1709788656.23 total 1315.31 last 26.93) - open/close 42859 (time 1709788666.26 total 1325.33 last 27.54) - open/close 43056 (time 1709788676.27 total 1335.35 last 19.67) - open/close 43241 (time 1709788686.27 total 1345.35 last 18.49) - open/close 43448 (time 1709788696.30 total 1355.38 last 20.64) - open/close 43698 (time 1709788706.32 total 1365.40 last 24.95) - open/close 43944 (time 1709788716.33 total 1375.40 last 24.59) - open/close 44198 (time 1709788726.34 total 1385.42 last 25.36) - open/close 44455 (time 1709788736.36 total 1395.44 last 25.65) - open/close 44699 (time 1709788746.39 total 1405.47 last 24.34) - open/close 44941 (time 1709788756.40 total 1415.48 last 24.18) - open/close 45226 (time 1709788766.42 total 1425.50 last 28.44) - open/close 45506 (time 1709788776.43 total 1435.50 last 27.99) - open/close 45746 (time 1709788786.45 total 1445.53 last 23.94) - open/close 46004 (time 1709788796.49 total 1455.57 last 25.70) - open/close 46255 (time 1709788806.51 total 1465.59 last 25.06) - open/close 46524 (time 1709788816.52 total 1475.59 last 26.89) - open/close 46790 (time 1709788826.54 total 1485.62 last 26.53) - open/close 47052 (time 1709788836.57 total 1495.64 last 26.14) - open/close 47300 (time 1709788846.58 total 1505.65 last 24.77) - open/close 47544 (time 1709788856.61 total 1515.68 last 24.33) - open/close 47824 (time 1709788866.62 total 1525.70 last 27.95) - open/close 48100 (time 1709788876.65 total 1535.72 last 27.53) - open/close 48364 (time 1709788886.67 total 1545.74 last 26.35) - open/close 48632 (time 1709788896.67 total 1555.75 last 26.79) - open/close 48890 (time 1709788906.69 total 1565.76 last 25.76) - open/close 49162 (time 1709788916.71 total 1575.78 last 27.16) - open/close 49412 (time 1709788926.71 total 1585.79 last 24.98) - open/close 49675 (time 1709788936.72 total 1595.80 last 26.28) - open/close 49925 (time 1709788946.75 total 1605.82 last 24.93) total: 50000 open/close in 1608.71 seconds: 31.08 ops/second - unlinked 0 (time 1709788950 ; total 0 ; last 0) - unlinked 10000 (time 1709789399 ; total 449 ; last 449) - unlinked 20000 (time 1709789739 ; total 789 ; last 340) - unlinked 30000 (time 1709789932 ; total 982 ; last 193) - unlinked 40000 (time 1709790018 ; total 1068 ; last 86) total: 50000 unlinks in 1106 seconds: 45.207958 unlinks/second umount lustre on /mnt/lustre..... Stopping client oleg415-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg415-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg415-server stop mds service on oleg415-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg415-server unloading modules on: 'oleg415-server' oleg415-server: oleg415-server.virtnet: executing unload_modules_local modules unloaded. pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-client: ssh exited with exit code 2 pdsh@oleg415-client: oleg415-server: ssh exited with exit code 2