== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 02:21:11 (1789366871) start mds service on oleg201-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg201-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg201-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg201-server: oleg201-server.virtnet: executing set_default_debug -1 all pdsh@oleg201-client: oleg201-server: ssh exited with exit code 1 Started lustre-OST0000 oleg201-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg201-client.virtnet: -o user_xattr,flock 192.168.202.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 582 (time 1789366917.79 total 10.02 last 58.10) - open/close/unlink 1210 (time 1789366927.81 total 20.03 last 62.73) - open/close/unlink 1772 (time 1789366937.81 total 30.04 last 56.16) - open/close/unlink 2273 (time 1789366947.82 total 40.04 last 50.09) - open/close/unlink 2792 (time 1789366957.83 total 50.05 last 51.84) - open/close/unlink 3363 (time 1789366967.83 total 60.05 last 57.09) - open/close/unlink 3916 (time 1789366977.84 total 70.06 last 55.26) - open/close/unlink 4523 (time 1789366987.84 total 80.06 last 60.67) - open/close/unlink 5192 (time 1789366997.84 total 90.07 last 66.87) - open/close/unlink 5909 (time 1789367007.85 total 100.08 last 71.63) - open/close/unlink 6456 (time 1789367017.86 total 110.08 last 54.69) - open/close/unlink 7070 (time 1789367027.87 total 120.09 last 61.33) - open/close/unlink 7687 (time 1789367037.88 total 130.10 last 61.64) - open/close/unlink 8361 (time 1789367047.89 total 140.11 last 67.32) - open/close/unlink 9028 (time 1789367057.89 total 150.11 last 66.70) - open/close/unlink 9732 (time 1789367067.89 total 160.11 last 70.38) - open/close/unlink 10000 (time 1789367072.77 total 165.00 last 54.90) - open/close/unlink 10672 (time 1789367082.78 total 175.00 last 67.16) - open/close/unlink 11361 (time 1789367092.79 total 185.01 last 68.85) - open/close/unlink 12116 (time 1789367102.81 total 195.03 last 75.37) - open/close/unlink 12742 (time 1789367112.82 total 205.04 last 62.53) - open/close/unlink 13396 (time 1789367122.83 total 215.05 last 65.33) - open/close/unlink 14122 (time 1789367132.83 total 225.05 last 72.60) - open/close/unlink 14801 (time 1789367142.83 total 235.06 last 67.85) - open/close/unlink 15428 (time 1789367152.85 total 245.07 last 62.59) - open/close/unlink 16031 (time 1789367162.87 total 255.09 last 60.20) - open/close/unlink 16636 (time 1789367172.87 total 265.09 last 60.49) - open/close/unlink 17250 (time 1789367182.89 total 275.11 last 61.27) - open/close/unlink 17725 (time 1789367192.91 total 285.13 last 47.42) - open/close/unlink 18419 (time 1789367202.92 total 295.14 last 69.34) - open/close/unlink 19047 (time 1789367212.92 total 305.14 last 62.77) - open/close/unlink 19772 (time 1789367222.93 total 315.15 last 72.46) - open/close/unlink 20000 (time 1789367227.15 total 319.38 last 53.94) - open/close/unlink 20674 (time 1789367237.17 total 329.39 last 67.32) - open/close/unlink 21368 (time 1789367247.17 total 339.40 last 69.34) - open/close/unlink 22045 (time 1789367257.20 total 349.42 last 67.54) - open/close/unlink 22749 (time 1789367267.20 total 359.42 last 70.40) - open/close/unlink 23466 (time 1789367277.22 total 369.44 last 71.54) - open/close/unlink 24017 (time 1789367287.23 total 379.46 last 55.02) - open/close/unlink 24702 (time 1789367297.25 total 389.48 last 68.37) - open/close/unlink 25260 (time 1789367307.27 total 399.49 last 55.70) - open/close/unlink 25881 (time 1789367317.27 total 409.50 last 62.08) - open/close/unlink 26510 (time 1789367327.28 total 419.50 last 62.89) - open/close/unlink 27067 (time 1789367337.29 total 429.52 last 55.59) - open/close/unlink 27577 (time 1789367347.30 total 439.53 last 50.95) - open/close/unlink 28168 (time 1789367357.31 total 449.54 last 59.04) - open/close/unlink 28800 (time 1789367367.32 total 459.54 last 63.18) - open/close/unlink 29487 (time 1789367377.33 total 469.55 last 68.64) - open/close/unlink 30000 (time 1789367385.04 total 477.26 last 66.52) - open/close/unlink 30621 (time 1789367395.06 total 487.28 last 61.98) - open/close/unlink 31210 (time 1789367405.07 total 497.29 last 58.85) - open/close/unlink 31914 (time 1789367415.07 total 507.29 last 70.39) - open/close/unlink 32521 (time 1789367425.07 total 517.29 last 60.69) - open/close/unlink 33139 (time 1789367435.08 total 527.30 last 61.75) - open/close/unlink 33862 (time 1789367445.08 total 537.30 last 72.29) - open/close/unlink 34678 (time 1789367455.09 total 547.31 last 81.56) - open/close/unlink 35587 (time 1789367465.09 total 557.31 last 90.88) - open/close/unli - open/close/unlink 585 (time 1789366917.63 total 10.01 last 58.45) - open/close/unlink 1210 (time 1789366927.64 total 20.01 last 62.48) - open/close/unlink 1771 (time 1789366937.65 total 30.02 last 56.04) - open/close/unlink 2274 (time 1789366947.66 total 40.03 last 50.26) - open/close/unlink 2788 (time 1789366957.66 total 50.03 last 51.39) - open/close/unlink 3359 (time 1789366967.66 total 60.03 last 57.10) - open/close/unlink 3912 (time 1789366977.66 total 70.04 last 55.28) - open/close/unlink 4524 (time 1789366987.67 total 80.05 last 61.12) - open/close/unlink 5182 (time 1789366997.70 total 90.07 last 65.66) - open/close/unlink 5901 (time 1789367007.70 total 100.07 last 71.89) - open/close/unlink 6443 (time 1789367017.70 total 110.08 last 54.19) - open/close/unlink 7049 (time 1789367027.71 total 120.09 last 60.51) - open/close/unlink 7664 (time 1789367037.71 total 130.09 last 61.50) - open/close/unlink 8337 (time 1789367047.71 total 140.09 last 67.29) - open/close/unlink 9003 (time 1789367057.73 total 150.10 last 66.53) - open/close/unlink 9710 (time 1789367067.73 total 160.11 last 70.66) - open/close/unlink 10000 (time 1789367072.96 total 165.33 last 55.51) - open/close/unlink 10666 (time 1789367082.96 total 175.34 last 66.56) - open/close/unlink 11351 (time 1789367092.96 total 185.34 last 68.50) - open/close/unlink 12104 (time 1789367102.98 total 195.36 last 75.16) - open/close/unlink 12724 (time 1789367112.99 total 205.37 last 61.92) - open/close/unlink 13375 (time 1789367122.99 total 215.37 last 65.09) - open/close/unlink 14094 (time 1789367133.02 total 225.39 last 71.74) - open/close/unlink 14783 (time 1789367143.02 total 235.40 last 68.86) - open/close/unlink 15414 (time 1789367153.02 total 245.40 last 63.10) - open/close/unlink 16011 (time 1789367163.03 total 255.41 last 59.64) - open/close/unlink 16618 (time 1789367173.05 total 265.43 last 60.59) - open/close/unlink 17222 (time 1789367183.06 total 275.43 last 60.36) - open/close/unlink 17711 (time 1789367193.06 total 285.44 last 48.86) - open/close/unlink 18410 (time 1789367203.07 total 295.44 last 69.89) - open/close/unlink 19030 (time 1789367213.07 total 305.45 last 61.98) - open/close/unlink 19746 (time 1789367223.08 total 315.45 last 71.55) - open/close/unlink 20000 (time 1789367227.75 total 320.13 last 54.36) - open/close/unlink 20677 (time 1789367237.76 total 330.13 last 67.65) - open/close/unlink 21371 (time 1789367247.77 total 340.15 last 69.32) - open/close/unlink 22041 (time 1789367257.78 total 350.16 last 66.90) - open/close/unlink 22757 (time 1789367267.79 total 360.17 last 71.55) - open/close/unlink 23456 (time 1789367277.79 total 370.17 last 69.90) - open/close/unlink 24011 (time 1789367287.79 total 380.17 last 55.50) - open/close/unlink 24702 (time 1789367297.81 total 390.19 last 68.97) - open/close/unlink 25249 (time 1789367307.83 total 400.20 last 54.61) - open/close/unlink 25882 (time 1789367317.83 total 410.20 last 63.30) - open/close/unlink 26496 (time 1789367327.85 total 420.22 last 61.28) - open/close/unlink 27045 (time 1789367337.85 total 430.23 last 54.88) - open/close/unlink 27570 (time 1789367347.86 total 440.24 last 52.44) - open/close/unlink 28166 (time 1789367357.86 total 450.24 last 59.60) - open/close/unlink 28801 (time 1789367367.89 total 460.27 last 63.32) - open/close/unlink 29497 (time 1789367377.90 total 470.27 last 69.55) - open/close/unlink 30000 (time 1789367385.61 total 477.98 last 65.25) - open/close/unlink 30607 (time 1789367395.61 total 487.99 last 60.66) - open/close/unlink 31188 (time 1789367405.62 total 497.99 last 58.08) - open/close/unlink 31913 (time 1789367415.62 total 507.99 last 72.49) - open/close/unlink 32509 (time 1789367425.63 total 518.01 last 59.50) - open/close/unlink 33133 (time 1789367435.63 total 528.01 last 62.40) - open/close/unlink 33854 (time 1789367445.65 total 538.02 last 72.00) - open/close/unlink 34688 (time 1789367455.65 total 548.02 last 83.39) - open/close/unlink 35596 (time 1789367465.66 total 558.04 last 90.67) - open/close/unli - open/close/unlink 580 (time 1789366917.88 total 10.01 last 57.97) - open/close/unlink 1209 (time 1789366927.89 total 20.02 last 62.84) - open/close/unlink 1765 (time 1789366937.89 total 30.02 last 55.60) - open/close/unlink 2270 (time 1789366947.90 total 40.03 last 50.45) - open/close/unlink 2798 (time 1789366957.91 total 50.04 last 52.71) - open/close/unlink 3366 (time 1789366967.93 total 60.06 last 56.70) - open/close/unlink 3912 (time 1789366977.94 total 70.07 last 54.55) - open/close/unlink 4517 (time 1789366987.96 total 80.09 last 60.37) - open/close/unlink 5184 (time 1789366997.97 total 90.09 last 66.69) - open/close/unlink 5902 (time 1789367007.97 total 100.10 last 71.77) - open/close/unlink 6446 (time 1789367017.98 total 110.11 last 54.34) - open/close/unlink 7058 (time 1789367027.98 total 120.11 last 61.20) - open/close/unlink 7680 (time 1789367037.99 total 130.11 last 62.16) - open/close/unlink 8368 (time 1789367048.00 total 140.13 last 68.70) - open/close/unlink 9043 (time 1789367058.01 total 150.14 last 67.43) - open/close/unlink 9751 (time 1789367068.04 total 160.17 last 70.63) - open/close/unlink 10000 (time 1789367072.44 total 164.57 last 56.57) - open/close/unlink 10663 (time 1789367082.44 total 174.57 last 66.29) - open/close/unlink 11345 (time 1789367092.46 total 184.59 last 68.07) - open/close/unlink 12118 (time 1789367102.47 total 194.60 last 77.22) - open/close/unlink 12739 (time 1789367112.48 total 204.60 last 62.06) - open/close/unlink 13389 (time 1789367122.49 total 214.62 last 64.90) - open/close/unlink 14106 (time 1789367132.50 total 224.63 last 71.65) - open/close/unlink 14794 (time 1789367142.51 total 234.64 last 68.69) - open/close/unlink 15420 (time 1789367152.52 total 244.65 last 62.57) - open/close/unlink 16019 (time 1789367162.52 total 254.65 last 59.89) - open/close/unlink 16612 (time 1789367172.53 total 264.66 last 59.26) - open/close/unlink 17232 (time 1789367182.54 total 274.67 last 61.90) - open/close/unlink 17698 (time 1789367192.55 total 284.68 last 46.56) - open/close/unlink 18387 (time 1789367202.56 total 294.69 last 68.83) - open/close/unlink 19016 (time 1789367212.57 total 304.70 last 62.86) - open/close/unlink 19738 (time 1789367222.57 total 314.70 last 72.20) - open/close/unlink 20000 (time 1789367227.42 total 319.54 last 54.07) - open/close/unlink 20677 (time 1789367237.42 total 329.55 last 67.68) - open/close/unlink 21369 (time 1789367247.42 total 339.55 last 69.16) - open/close/unlink 22041 (time 1789367257.43 total 349.56 last 67.15) - open/close/unlink 22751 (time 1789367267.43 total 359.56 last 70.98) - open/close/unlink 23459 (time 1789367277.44 total 369.57 last 70.77) - open/close/unlink 24013 (time 1789367287.45 total 379.58 last 55.35) - open/close/unlink 24705 (time 1789367297.46 total 389.59 last 69.11) - open/close/unlink 25257 (time 1789367307.46 total 399.59 last 55.19) - open/close/unlink 25889 (time 1789367317.47 total 409.60 last 63.13) - open/close/unlink 26506 (time 1789367327.49 total 419.61 last 61.62) - open/close/unlink 27052 (time 1789367337.52 total 429.65 last 54.39) - open/close/unlink 27561 (time 1789367347.53 total 439.66 last 50.86) - open/close/unlink 28145 (time 1789367357.54 total 449.67 last 58.36) - open/close/unlink 28775 (time 1789367367.55 total 459.68 last 62.95) - open/close/unlink 29472 (time 1789367377.56 total 469.68 last 69.65) - open/close/unlink 30000 (time 1789367385.61 total 477.74 last 65.54) - open/close/unlink 30605 (time 1789367395.61 total 487.74 last 60.49) - open/close/unlink 31196 (time 1789367405.63 total 497.76 last 59.02) - open/close/unlink 31914 (time 1789367415.63 total 507.76 last 71.78) - open/close/unlink 32502 (time 1789367425.64 total 517.77 last 58.73) - open/close/unlink 33121 (time 1789367435.65 total 527.78 last 61.84) - open/close/unlink 33846 (time 1789367445.66 total 537.79 last 72.41) - open/close/unlink 34674 (time 1789367455.68 total 547.80 last 82.69) - open/close/unlink 35582 (time 1789367465.68 total 557.81 last 90.78) - open/close/unli - open/close/unlink 585 (time 1789366917.94 total 10.02 last 58.36) - open/close/unlink 1218 (time 1789366927.94 total 20.03 last 63.29) - open/close/unlink 1774 (time 1789366937.96 total 30.04 last 55.50) - open/close/unlink 2274 (time 1789366947.97 total 40.06 last 49.94) - open/close/unlink 2801 (time 1789366957.98 total 50.06 last 52.69) - open/close/unlink 3371 (time 1789366967.99 total 60.07 last 56.93) - open/close/unlink 3922 (time 1789366977.99 total 70.07 last 55.08) - open/close/unlink 4528 (time 1789366988.00 total 80.09 last 60.54) - open/close/unlink 5201 (time 1789366998.01 total 90.09 last 67.23) - open/close/unlink 5921 (time 1789367008.02 total 100.10 last 71.93) - open/close/unlink 6458 (time 1789367018.03 total 110.11 last 53.66) - open/close/unlink 7066 (time 1789367028.05 total 120.14 last 60.66) - open/close/unlink 7694 (time 1789367038.05 total 130.14 last 62.79) - open/close/unlink 8373 (time 1789367048.07 total 140.15 last 67.83) - open/close/unlink 9044 (time 1789367058.08 total 150.16 last 67.02) - open/close/unlink 9742 (time 1789367068.08 total 160.16 last 69.79) - open/close/unlink 10000 (time 1789367072.78 total 164.86 last 54.86) - open/close/unlink 10662 (time 1789367082.79 total 174.87 last 66.14) - open/close/unlink 11355 (time 1789367092.79 total 184.87 last 69.28) - open/close/unlink 12111 (time 1789367102.80 total 194.88 last 75.55) - open/close/unlink 12730 (time 1789367112.80 total 204.88 last 61.88) - open/close/unlink 13384 (time 1789367122.81 total 214.89 last 65.36) - open/close/unlink 14110 (time 1789367132.82 total 224.90 last 72.54) - open/close/unlink 14788 (time 1789367142.83 total 234.91 last 67.70) - open/close/unlink 15418 (time 1789367152.83 total 244.91 last 62.98) - open/close/unlink 16020 (time 1789367162.84 total 254.92 last 60.14) - open/close/unlink 16616 (time 1789367172.84 total 264.92 last 59.60) - open/close/unlink 17217 (time 1789367182.84 total 274.93 last 60.10) - open/close/unlink 17688 (time 1789367192.85 total 284.93 last 47.07) - open/close/unlink 18376 (time 1789367202.86 total 294.94 last 68.76) - open/close/unlink 19007 (time 1789367212.87 total 304.95 last 63.00) - open/close/unlink 19726 (time 1789367222.87 total 314.96 last 71.88) - open/close/unlink 20000 (time 1789367227.80 total 319.88 last 55.65) - open/close/unlink 20676 (time 1789367237.80 total 329.88 last 67.57) - open/close/unlink 21376 (time 1789367247.82 total 339.90 last 69.87) - open/close/unlink 22048 (time 1789367257.82 total 349.90 last 67.20) - open/close/unlink 22771 (time 1789367267.82 total 359.91 last 72.29) - open/close/unlink 23469 (time 1789367277.83 total 369.91 last 69.76) - open/close/unlink 24025 (time 1789367287.84 total 379.92 last 55.53) - open/close/unlink 24711 (time 1789367297.85 total 389.93 last 68.56) - open/close/unlink 25264 (time 1789367307.88 total 399.96 last 55.12) - open/close/unlink 25900 (time 1789367317.88 total 409.97 last 63.59) - open/close/unlink 26514 (time 1789367327.90 total 419.98 last 61.32) - open/close/unlink 27060 (time 1789367337.92 total 430.00 last 54.48) - open/close/unlink 27582 (time 1789367347.93 total 440.01 last 52.16) - open/close/unlink 28168 (time 1789367357.94 total 450.02 last 58.53) - open/close/unlink 28797 (time 1789367367.94 total 460.03 last 62.87) - open/close/unlink 29501 (time 1789367377.94 total 470.03 last 70.40) - open/close/unlink 30000 (time 1789367385.67 total 477.76 last 64.55) - open/close/unlink 30609 (time 1789367395.68 total 487.76 last 60.87) - open/close/unlink 31198 (time 1789367405.68 total 497.76 last 58.88) - open/close/unlink 31924 (time 1789367415.69 total 507.77 last 72.55) - open/close/unlink 32512 (time 1789367425.70 total 517.78 last 58.76) - open/close/unlink 33127 (time 1789367435.71 total 527.79 last 61.42) - open/close/unlink 33850 (time 1789367445.72 total 537.80 last 72.24) - open/close/unlink 34681 (time 1789367455.73 total 547.81 last 82.98) - open/close/unlink 35569 (time 1789367465.75 total 557.83 last 88.62) - open/close/unli - open/close/unlink 582 (time 1789366917.98 total 10.01 last 58.13) - open/close/unlink 1223 (time 1789366928.00 total 20.03 last 64.00) - open/close/unlink 1778 (time 1789366938.00 total 30.03 last 55.49) - open/close/unlink 2284 (time 1789366948.02 total 40.05 last 50.51) - open/close/unlink 2815 (time 1789366958.03 total 50.06 last 53.06) - open/close/unlink 3382 (time 1789366968.05 total 60.08 last 56.57) - open/close/unlink 3934 (time 1789366978.06 total 70.08 last 55.17) - open/close/unlink 4532 (time 1789366988.07 total 80.10 last 59.70) - open/close/unlink 5209 (time 1789366998.08 total 90.11 last 67.66) - open/close/unlink 5923 (time 1789367008.10 total 100.12 last 71.29) - open/close/unlink 6459 (time 1789367018.11 total 110.14 last 53.50) - open/close/unlink 7058 (time 1789367028.13 total 120.16 last 59.80) - open/close/unlink 7687 (time 1789367038.14 total 130.17 last 62.86) - open/close/unlink 8355 (time 1789367048.14 total 140.17 last 66.76) - open/close/unlink 9025 (time 1789367058.16 total 150.19 last 66.88) - open/close/unlink 9718 (time 1789367068.16 total 160.19 last 69.30) - open/close/unlink 10000 (time 1789367073.20 total 165.22 last 55.99) - open/close/unlink 10669 (time 1789367083.20 total 175.23 last 66.85) - open/close/unlink 11351 (time 1789367093.21 total 185.23 last 68.19) - open/close/unlink 12095 (time 1789367103.21 total 195.24 last 74.34) - open/close/unlink 12722 (time 1789367113.23 total 205.26 last 62.57) - open/close/unlink 13368 (time 1789367123.25 total 215.28 last 64.51) - open/close/unlink 14091 (time 1789367133.27 total 225.29 last 72.17) - open/close/unlink 14774 (time 1789367143.27 total 235.30 last 68.24) - open/close/unlink 15404 (time 1789367153.29 total 245.32 last 62.91) - open/close/unlink 16004 (time 1789367163.30 total 255.33 last 59.91) - open/close/unlink 16620 (time 1789367173.32 total 265.34 last 61.52) - open/close/unlink 17209 (time 1789367183.33 total 275.36 last 58.81) - open/close/unlink 17704 (time 1789367193.33 total 285.36 last 49.49) - open/close/unlink 18408 (time 1789367203.34 total 295.36 last 70.39) - open/close/unlink 19023 (time 1789367213.34 total 305.37 last 61.45) - open/close/unlink 19747 (time 1789367223.35 total 315.38 last 72.34) - open/close/unlink 20000 (time 1789367227.93 total 319.96 last 55.24) - open/close/unlink 20673 (time 1789367237.94 total 329.97 last 67.24) - open/close/unlink 21359 (time 1789367247.95 total 339.98 last 68.54) - open/close/unlink 22027 (time 1789367257.96 total 349.99 last 66.70) - open/close/unlink 22756 (time 1789367267.97 total 360.00 last 72.85) - open/close/unlink 23453 (time 1789367277.98 total 370.01 last 69.66) - open/close/unlink 24011 (time 1789367287.98 total 380.01 last 55.78) - open/close/unlink 24689 (time 1789367298.00 total 390.03 last 67.68) - open/close/unlink 25236 (time 1789367308.01 total 400.03 last 54.66) - open/close/unlink 25878 (time 1789367318.02 total 410.04 last 64.13) - open/close/unlink 26491 (time 1789367328.02 total 420.05 last 61.29) - open/close/unlink 27034 (time 1789367338.05 total 430.08 last 54.13) - open/close/unlink 27552 (time 1789367348.05 total 440.08 last 51.80) - open/close/unlink 28159 (time 1789367358.05 total 450.08 last 60.69) - open/close/unlink 28780 (time 1789367368.06 total 460.08 last 62.07) - open/close/unlink 29492 (time 1789367378.06 total 470.09 last 71.17) - open/close/unlink 30000 (time 1789367386.24 total 478.27 last 62.11) - open/close/unlink 30607 (time 1789367396.25 total 488.27 last 60.67) - open/close/unlink 31207 (time 1789367406.25 total 498.28 last 59.99) - open/close/unlink 31930 (time 1789367416.25 total 508.28 last 72.30) - open/close/unlink 32512 (time 1789367426.26 total 518.29 last 58.11) - open/close/unlink 33129 (time 1789367436.27 total 528.29 last 61.69) - open/close/unlink 33857 (time 1789367446.27 total 538.30 last 72.77) - open/close/unlink 34714 (time 1789367456.27 total 548.30 last 85.67) - open/close/unlink 35585 (time 1789367466.28 total 558.31 last 87.02) - open/close/unlink 36326 (time 1789367475.68 total 567.81 last 74.39) - open/close/unlink 37228 (time 1789367485.68 total 577.81 last 90.15) - open/close/unlink 37949 (time 1789367495.70 total 587.83 last 71.98) - open/close/unlink 38851 (time 1789367505.70 total 597.83 last 90.20) - open/close/unlink 39708 (time 1789367515.70 total 607.83 last 85.69) - open/close/unlink 40000 (time 1789367519.81 total 611.93 last 71.13) - open/close/unlink 40651 (time 1789367529.81 total 621.94 last 65.08) - open/close/unlink 41347 (time 1789367539.82 total 631.94 last 69.56) - open/close/unlink 42005 (time 1789367549.83 total 641.96 last 65.72) - open/close/unlink 42706 (time 1789367559.83 total 651.96 last 70.06) - open/close/unlink 43395 (time 1789367569.84 total 661.97 last 68.84) - open/close/unlink 44308 (time 1789367579.85 total 671.98 last 91.26) - open/close/unlink 45023 (time 1789367589.85 total 681.98 last 71.46) - open/close/unlink 45800 (time 1789367599.86 total 691.99 last 77.64) - open/close/unlink 46411 (time 1789367609.87 total 701.99 last 61.06) - open/close/unlink 47035 (time 1789367619.88 total 712.01 last 62.31) - open/close/unlink 47690 (time 1789367629.90 total 722.03 last 65.38) - open/close/unlink 48516 (time 1789367639.91 total 732.04 last 82.51) - open/close/unlink 49235 (time 1789367649.91 total 742.04 last 71.87) - open/close/unlink 50000 (time 1789367659.91 total 752.04 last 76.52) - open/close/unlink 50698 (time 1789367669.92 total 762.05 last 69.75) - open/close/unlink 51450 (time 1789367679.92 total 772.05 last 75.18) - open/close/unlink 52177 (time 1789367689.93 total 782.05 last 72.67) - open/close/unlink 52964 (time 1789367699.93 total 792.06 last 78.62) - open/close/unlink 53852 (time 1789367709.94 total 802.07 last 88.77) - open/close/unlink 54686 (time 1789367719.94 total 812.07 last 83.36) - open/close/unlink 55479 (time 1789367729.96 total 822.09 last 79.17) - open/close/unlink 56292 (time 1789367739.96 total 832.09 last 81.29) - open/close/unlink 57081 (time 1789367749.97 total 842.10 last 78.84) - open/close/unlink 57919 (time 1789367759.97 total 852.10 last 83.74) - open/close/unlink 58539 (time 1789367769.98 total 862.11 last 61.95) - open/close/unlink 59319 (time 1789367779.99 total 872.11 last 77.98) - open/close/unlink 60000 (time 1789367789.35 total 881.47 last 72.75) - open/close/unlink 60839 (time 1789367799.36 total 891.48 last 83.82) - open/close/unlink 61628 (time 1789367809.36 total 901.49 last 78.85) - open/close/unlink 62383 (time 1789367819.38 total 911.51 last 75.37) - open/close/unlink 62930 (time 1789367829.41 total 921.54 last 54.53) - open/close/unlink 63522 (time 1789367839.43 total 931.55 last 59.11) - open/close/unlink 64186 (time 1789367849.43 total 941.56 last 66.38) total: 64768 open/close/unlink in 951.50 seconds: 68.07 ops/second nk 36333 (time 1789367475.67 total 568.05 last 73.66) - open/close/unlink 37242 (time 1789367485.68 total 578.05 last 90.81) - open/close/unlink 37961 (time 1789367495.68 total 588.06 last 71.87) - open/close/unlink 38849 (time 1789367505.69 total 598.06 last 88.75) - open/close/unlink 39707 (time 1789367515.69 total 608.07 last 85.73) - open/close/unlink 40000 (time 1789367519.79 total 612.16 last 71.59) - open/close/unlink 40644 (time 1789367529.79 total 622.17 last 64.38) - open/close/unlink 41326 (time 1789367539.79 total 632.17 last 68.16) - open/close/unlink 41980 (time 1789367549.80 total 642.17 last 65.39) - open/close/unlink 42683 (time 1789367559.80 total 652.18 last 70.28) - open/close/unlink 43376 (time 1789367569.81 total 662.19 last 69.21) - open/close/unlink 44282 (time 1789367579.82 total 672.19 last 90.57) - open/close/unlink 44992 (time 1789367589.83 total 682.20 last 70.93) - open/close/unlink 45767 (time 1789367599.83 total 692.20 last 77.49) - open/close/unlink 46387 (time 1789367609.84 total 702.22 last 61.92) - open/close/unlink 47014 (time 1789367619.85 total 712.23 last 62.64) - open/close/unlink 47667 (time 1789367629.86 total 722.24 last 65.22) - open/close/unlink 48495 (time 1789367639.87 total 732.24 last 82.75) - open/close/unlink 49215 (time 1789367649.87 total 742.25 last 71.95) - open/close/unlink 49973 (time 1789367659.88 total 752.25 last 75.78) - open/close/unlink 50671 (time 1789367669.90 total 762.28 last 69.64) - open/close/unlink 51411 (time 1789367679.92 total 772.29 last 73.88) - open/close/unlink 52132 (time 1789367689.92 total 782.30 last 72.07) - open/close/unlink 52912 (time 1789367699.92 total 792.30 last 77.97) - open/close/unlink 53800 (time 1789367709.93 total 802.31 last 88.70) - open/close/unlink 54640 (time 1789367719.94 total 812.32 last 83.94) - open/close/unlink 55433 (time 1789367729.96 total 822.34 last 79.15) - open/close/unlink 56253 (time 1789367739.96 total 832.34 last 82.00) - open/close/unlink 57037 (time 1789367749.96 total 842.34 last 78.39) - open/close/unlink 57879 (time 1789367759.97 total 852.34 last 84.16) - open/close/unlink 58505 (time 1789367769.97 total 862.35 last 62.56) - open/close/unlink 59279 (time 1789367779.98 total 872.36 last 77.31) - open/close/unlink 60000 (time 1789367789.77 total 882.15 last 73.69) - open/close/unlink 60822 (time 1789367799.77 total 892.15 last 82.17) - open/close/unlink 61620 (time 1789367809.78 total 902.15 last 79.76) - open/close/unlink 62357 (time 1789367819.80 total 912.18 last 73.54) - open/close/unlink 62908 (time 1789367829.80 total 922.18 last 55.09) - open/close/unlink 63498 (time 1789367839.81 total 932.18 last 58.95) - open/close/unlink 64168 (time 1789367849.82 total 942.19 last 66.95) - open/close/unlink 64741 (time 1789367859.83 total 952.21 last 57.20) total: 64768 open/close/unlink in 953.08 seconds: 67.96 ops/second nk 36306 (time 1789367475.10 total 567.32 last 71.81) - open/close/unlink 37226 (time 1789367485.11 total 577.33 last 91.92) - open/close/unlink 37949 (time 1789367495.11 total 587.34 last 72.26) - open/close/unlink 38844 (time 1789367505.12 total 597.34 last 89.47) - open/close/unlink 39706 (time 1789367515.12 total 607.35 last 86.14) - open/close/unlink 40000 (time 1789367519.32 total 611.54 last 70.11) - open/close/unlink 40643 (time 1789367529.33 total 621.55 last 64.22) - open/close/unlink 41335 (time 1789367539.35 total 631.57 last 69.10) - open/close/unlink 41990 (time 1789367549.36 total 641.58 last 65.41) - open/close/unlink 42697 (time 1789367559.37 total 651.59 last 70.64) - open/close/unlink 43386 (time 1789367569.38 total 661.61 last 68.80) - open/close/unlink 44283 (time 1789367579.40 total 671.62 last 89.58) - open/close/unlink 44998 (time 1789367589.40 total 681.62 last 71.46) - open/close/unlink 45761 (time 1789367599.42 total 691.64 last 76.20) - open/close/unlink 46394 (time 1789367609.42 total 701.64 last 63.29) - open/close/unlink 47016 (time 1789367619.42 total 711.64 last 62.17) - open/close/unlink 47665 (time 1789367629.42 total 721.65 last 64.89) - open/close/unlink 48487 (time 1789367639.44 total 731.66 last 82.09) - open/close/unlink 49204 (time 1789367649.44 total 741.66 last 71.67) - open/close/unlink 49952 (time 1789367659.44 total 751.66 last 74.80) - open/close/unlink 50660 (time 1789367669.44 total 761.66 last 70.79) - open/close/unlink 51394 (time 1789367679.45 total 771.67 last 73.34) - open/close/unlink 52105 (time 1789367689.45 total 781.67 last 71.09) - open/close/unlink 52877 (time 1789367699.46 total 791.68 last 77.15) - open/close/unlink 53758 (time 1789367709.46 total 801.69 last 88.06) - open/close/unlink 54605 (time 1789367719.47 total 811.69 last 84.66) - open/close/unlink 55397 (time 1789367729.47 total 821.70 last 79.15) - open/close/unlink 56206 (time 1789367739.48 total 831.71 last 80.82) - open/close/unlink 57002 (time 1789367749.49 total 841.71 last 79.54) - open/close/unlink 57828 (time 1789367759.50 total 851.72 last 82.57) - open/close/unlink 58469 (time 1789367769.50 total 861.72 last 64.06) - open/close/unlink 59247 (time 1789367779.50 total 871.73 last 77.79) - open/close/unlink 59977 (time 1789367789.50 total 881.73 last 72.99) - open/close/unlink 60810 (time 1789367799.51 total 891.73 last 83.25) - open/close/unlink 61597 (time 1789367809.52 total 901.75 last 78.60) - open/close/unlink 62334 (time 1789367819.54 total 911.76 last 73.60) - open/close/unlink 62880 (time 1789367829.55 total 921.77 last 54.54) - open/close/unlink 63465 (time 1789367839.56 total 931.78 last 58.43) - open/close/unlink 64125 (time 1789367849.57 total 941.79 last 65.93) - open/close/unlink 64698 (time 1789367859.57 total 951.79 last 57.30) total: 64768 open/close/unlink in 953.93 seconds: 67.90 ops/second nk 36333 (time 1789367476.29 total 568.31 last 74.78) - open/close/unlink 37231 (time 1789367486.29 total 578.32 last 89.77) - open/close/unlink 37953 (time 1789367496.30 total 588.32 last 72.14) - open/close/unlink 38856 (time 1789367506.30 total 598.33 last 90.26) - open/close/unlink 39723 (time 1789367516.31 total 608.34 last 86.63) - open/close/unlink 40000 (time 1789367520.35 total 612.38 last 68.59) - open/close/unlink 40647 (time 1789367530.35 total 622.38 last 64.66) - open/close/unlink 41343 (time 1789367540.36 total 632.39 last 69.54) - open/close/unlink 42006 (time 1789367550.37 total 642.40 last 66.26) - open/close/unlink 42699 (time 1789367560.38 total 652.40 last 69.25) - open/close/unlink 43382 (time 1789367570.38 total 662.41 last 68.28) - open/close/unlink 44287 (time 1789367580.38 total 672.41 last 90.44) - open/close/unlink 44991 (time 1789367590.39 total 682.41 last 70.40) - open/close/unlink 45768 (time 1789367600.40 total 692.43 last 77.60) - open/close/unlink 46375 (time 1789367610.40 total 702.43 last 60.69) - open/close/unlink 47013 (time 1789367620.41 total 712.44 last 63.75) - open/close/unlink 47677 (time 1789367630.42 total 722.44 last 66.35) - open/close/unlink 48493 (time 1789367640.42 total 732.45 last 81.58) - open/close/unlink 49210 (time 1789367650.43 total 742.46 last 71.58) - open/close/unlink 49979 (time 1789367660.44 total 752.47 last 76.83) - open/close/unlink 50666 (time 1789367670.45 total 762.48 last 68.66) - open/close/unlink 51413 (time 1789367680.46 total 772.48 last 74.66) - open/close/unlink 52135 (time 1789367690.46 total 782.49 last 72.17) - open/close/unlink 52931 (time 1789367700.48 total 792.51 last 79.46) - open/close/unlink 53819 (time 1789367710.48 total 802.51 last 88.76) - open/close/unlink 54655 (time 1789367720.48 total 812.51 last 83.58) - open/close/unlink 55443 (time 1789367730.49 total 822.51 last 78.79) - open/close/unlink 56276 (time 1789367740.49 total 832.52 last 83.27) - open/close/unlink 57056 (time 1789367750.49 total 842.52 last 77.96) - open/close/unlink 57913 (time 1789367760.50 total 852.53 last 85.65) - open/close/unlink 58525 (time 1789367770.51 total 862.53 last 61.17) - open/close/unlink 59311 (time 1789367780.52 total 872.55 last 78.46) - open/close/unlink 60000 (time 1789367789.96 total 881.99 last 73.00) - open/close/unlink 60818 (time 1789367799.97 total 892.00 last 81.73) - open/close/unlink 61611 (time 1789367809.98 total 902.01 last 79.20) - open/close/unlink 62341 (time 1789367819.99 total 912.02 last 72.93) - open/close/unlink 62883 (time 1789367829.99 total 922.02 last 54.19) - open/close/unlink 63465 (time 1789367840.00 total 932.03 last 58.15) - open/close/unlink 64132 (time 1789367850.00 total 942.03 last 66.69) - open/close/unlink 64700 (time 1789367860.16 total 952.19 last 55.94) total: 64768 open/close/unlink in 954.06 seconds: 67.89 ops/second nk 36308 (time 1789367475.75 total 567.83 last 73.90) - open/close/unlink 37206 (time 1789367485.76 total 577.84 last 89.72) - open/close/unlink 37926 (time 1789367495.77 total 587.85 last 71.94) - open/close/unlink 38822 (time 1789367505.77 total 597.85 last 89.58) - open/close/unlink 39684 (time 1789367515.78 total 607.86 last 86.15) - open/close/unlink 40000 (time 1789367520.30 total 612.38 last 69.92) - open/close/unlink 40638 (time 1789367530.30 total 622.38 last 63.78) - open/close/unlink 41322 (time 1789367540.31 total 632.39 last 68.34) - open/close/unlink 41973 (time 1789367550.31 total 642.39 last 65.09) - open/close/unlink 42667 (time 1789367560.32 total 652.40 last 69.35) - open/close/unlink 43360 (time 1789367570.33 total 662.41 last 69.21) - open/close/unlink 44272 (time 1789367580.34 total 672.42 last 91.13) - open/close/unlink 44979 (time 1789367590.36 total 682.44 last 70.55) - open/close/unlink 45758 (time 1789367600.37 total 692.46 last 77.80) - open/close/unlink 46372 (time 1789367610.39 total 702.47 last 61.31) - open/close/unlink 47013 (time 1789367620.39 total 712.47 last 64.08) - open/close/unlink 47671 (time 1789367630.39 total 722.48 last 65.79) - open/close/unlink 48481 (time 1789367640.40 total 732.48 last 80.97) - open/close/unlink 49197 (time 1789367650.41 total 742.49 last 71.53) - open/close/unlink 49975 (time 1789367660.41 total 752.49 last 77.78) - open/close/unlink 50660 (time 1789367670.42 total 762.50 last 68.43) - open/close/unlink 51406 (time 1789367680.43 total 772.51 last 74.53) - open/close/unlink 52126 (time 1789367690.43 total 782.51 last 71.98) - open/close/unlink 52921 (time 1789367700.44 total 792.52 last 79.43) - open/close/unlink 53799 (time 1789367710.44 total 802.53 last 87.79) - open/close/unlink 54634 (time 1789367720.45 total 812.53 last 83.48) - open/close/unlink 55409 (time 1789367730.45 total 822.53 last 77.47) - open/close/unlink 56232 (time 1789367740.46 total 832.54 last 82.25) - open/close/unlink 56997 (time 1789367750.46 total 842.55 last 76.45) - open/close/unlink 57850 (time 1789367760.47 total 852.55 last 85.26) - open/close/unlink 58464 (time 1789367770.48 total 862.57 last 61.31) - open/close/unlink 59258 (time 1789367780.51 total 872.59 last 79.19) - open/close/unlink 59991 (time 1789367790.52 total 882.60 last 73.22) - open/close/unlink 60819 (time 1789367800.52 total 892.60 last 82.79) - open/close/unlink 61615 (time 1789367810.52 total 902.60 last 79.59) - open/close/unlink 62324 (time 1789367820.52 total 912.61 last 70.89) - open/close/unlink 62872 (time 1789367830.53 total 922.61 last 54.76) - open/close/unlink 63478 (time 1789367840.53 total 932.61 last 60.59) - open/close/unlink 64140 (time 1789367850.55 total 942.63 last 66.09) - open/close/unlink 64691 (time 1789367860.56 total 952.64 last 55.06) total: 64768 open/close/unlink in 954.84 seconds: 67.83 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg201-server: cannot open 'lustre-ost2': no such pool pdsh@oleg201-client: oleg201-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg201-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg201-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg201-server stop mds service on oleg201-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg201-server unloading modules via unload_modules_local on: 'oleg201-server' oleg201-server: oleg201-server.virtnet: executing unload_modules_local oleg201-server: kmemleak disabled modules unloaded. pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-client: ssh exited with exit code 2 pdsh@oleg201-client: oleg201-server: ssh exited with exit code 2