== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:54:42 (1779310482) start mds service on oleg409-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg409-server Start mds2: mount -t lustre -o localrecov lustre-mdt2/mdt2 /mnt/lustre-mds2 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg409-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg409-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg409-server: oleg409-server.virtnet: executing set_default_debug -1 all pdsh@oleg409-client: oleg409-server: ssh exited with exit code 1 Started lustre-OST0000 oleg409-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg409-client.virtnet: -o user_xattr,flock 192.168.204.109@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92602 331 92271 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 92870 312 92558 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 12395 491 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12547 643 11904 6% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 499 (time 1779310537.38 total 10.05 last 49.65) - open/close/unlink 971 (time 1779310547.39 total 20.06 last 47.14) - open/close/unlink 1418 (time 1779310557.41 total 30.09 last 44.61) - open/close/unlink 1896 (time 1779310567.42 total 40.09 last 47.77) - open/close/unlink 2303 (time 1779310577.42 total 50.10 last 40.66) - open/close/unlink 2679 (time 1779310587.44 total 60.11 last 37.56) - open/close/unlink 3048 (time 1779310597.44 total 70.12 last 36.88) - open/close/unlink 3478 (time 1779310607.44 total 80.12 last 42.99) - open/close/unlink 3872 (time 1779310617.47 total 90.14 last 39.31) - open/close/unlink 4279 (time 1779310627.47 total 100.14 last 40.69) - open/close/unlink 4736 (time 1779310637.49 total 110.17 last 45.60) - open/close/unlink 5139 (time 1779310647.52 total 120.20 last 40.17) - open/close/unlink 5543 (time 1779310657.53 total 130.20 last 40.39) - open/close/unlink 5907 (time 1779310667.53 total 140.20 last 36.40) - open/close/unlink 6267 (time 1779310677.56 total 150.23 last 35.90) - open/close/unlink 6611 (time 1779310687.58 total 160.25 last 34.33) - open/close/unlink 7038 (time 1779310697.58 total 170.25 last 42.68) - open/close/unlink 7479 (time 1779310707.59 total 180.26 last 44.06) - open/close/unlink 7913 (time 1779310717.62 total 190.30 last 43.25) - open/close/unlink 8362 (time 1779310727.66 total 200.34 last 44.71) - open/close/unlink 8800 (time 1779310737.69 total 210.36 last 43.70) - open/close/unlink 9314 (time 1779310747.69 total 220.37 last 51.37) - open/close/unlink 9717 (time 1779310757.71 total 230.38 last 40.24) - open/close/unlink 10000 (time 1779310765.10 total 237.78 last 38.26) - open/close/unlink 10418 (time 1779310775.11 total 247.79 last 41.76) - open/close/unlink 10993 (time 1779310785.11 total 257.79 last 57.48) - open/close/unlink 11675 (time 1779310795.13 total 267.80 last 68.11) - open/close/unlink 12193 (time 1779310805.14 total 277.82 last 51.73) - open/close/unlink 12638 (time 1779310815.17 total 287.84 last 44.39) - open/close/unlink 13181 (time 1779310825.17 total 297.84 last 54.30) - open/close/unlink 13756 (time 1779310835.18 total 307.85 last 57.45) - open/close/unlink 14348 (time 1779310845.18 total 317.86 last 59.16) - open/close/unlink 14877 (time 1779310855.21 total 327.89 last 52.76) - open/close/unlink 15407 (time 1779310865.21 total 337.89 last 53.00) - open/close/unlink 15998 (time 1779310875.22 total 347.89 last 59.06) - open/close/unlink 16535 (time 1779310885.23 total 357.90 last 53.66) - open/close/unlink 17028 (time 1779310895.24 total 367.91 last 49.24) - open/close/unlink 17521 (time 1779310905.25 total 377.93 last 49.24) - open/close/unlink 17983 (time 1779310915.26 total 387.94 last 46.14) - open/close/unlink 18541 (time 1779310925.26 total 397.94 last 55.80) - open/close/unlink 19042 (time 1779310935.27 total 407.95 last 50.06) - open/close/unlink 19598 (time 1779310945.27 total 417.95 last 55.59) - open/close/unlink 20000 (time 1779310951.94 total 424.61 last 60.33) - open/close/unlink 20576 (time 1779310961.94 total 434.61 last 57.58) - open/close/unlink 21156 (time 1779310971.95 total 444.62 last 57.97) - open/close/unlink 21687 (time 1779310981.95 total 454.62 last 53.09) - open/close/unlink 22267 (time 1779310991.95 total 464.63 last 57.95) - open/close/unlink 22798 (time 1779311001.98 total 474.65 last 52.98) - open/close/unlink 23313 (time 1779311011.98 total 484.65 last 51.49) - open/close/unlink 23812 (time 1779311021.99 total 494.67 last 49.82) - open/close/unlink 24336 (time 1779311032.01 total 504.69 last 52.32) - open/close/unlink 24862 (time 1779311042.02 total 514.69 last 52.57) - open/close/unlink 25408 (time 1779311052.03 total 524.71 last 54.52) - open/close/unlink 25990 (time 1779311062.04 total 534.71 last 58.16) - open/close/unlink 26670 (time 1779311072.04 total 544.72 last 67.98) - open/close/unlink 27404 (time 1779311082.05 total 554.72 last 73.34) - open/close/unlink 28118 (time 1779311092.05 total 564.73 last 71.39) - open/close/unlink 28824 - open/close/unlink 490 (time 1779310536.95 total 10.00 last 48.98) - open/close/unlink 968 (time 1779310546.95 total 20.01 last 47.79) - open/close/unlink 1417 (time 1779310556.96 total 30.02 last 44.84) - open/close/unlink 1897 (time 1779310566.98 total 40.04 last 47.92) - open/close/unlink 2304 (time 1779310576.98 total 50.04 last 40.69) - open/close/unlink 2671 (time 1779310587.00 total 60.05 last 36.65) - open/close/unlink 3042 (time 1779310597.00 total 70.06 last 37.10) - open/close/unlink 3475 (time 1779310607.02 total 80.07 last 43.22) - open/close/unlink 3865 (time 1779310617.04 total 90.09 last 38.92) - open/close/unlink 4273 (time 1779310627.04 total 100.10 last 40.77) - open/close/unlink 4729 (time 1779310637.07 total 110.12 last 45.49) - open/close/unlink 5144 (time 1779310647.08 total 120.13 last 41.46) - open/close/unlink 5533 (time 1779310657.08 total 130.14 last 38.87) - open/close/unlink 5907 (time 1779310667.08 total 140.14 last 37.39) - open/close/unlink 6281 (time 1779310677.10 total 150.16 last 37.34) - open/close/unlink 6611 (time 1779310687.12 total 160.18 last 32.93) - open/close/unlink 7038 (time 1779310697.14 total 170.20 last 42.62) - open/close/unlink 7475 (time 1779310707.15 total 180.21 last 43.63) - open/close/unlink 7909 (time 1779310717.16 total 190.22 last 43.37) - open/close/unlink 8359 (time 1779310727.17 total 200.23 last 44.97) - open/close/unlink 8805 (time 1779310737.18 total 210.24 last 44.55) - open/close/unlink 9312 (time 1779310747.18 total 220.24 last 50.69) - open/close/unlink 9724 (time 1779310757.19 total 230.25 last 41.17) - open/close/unlink 10000 (time 1779310764.74 total 237.79 last 36.58) - open/close/unlink 10404 (time 1779310774.75 total 247.80 last 40.36) - open/close/unlink 10985 (time 1779310784.76 total 257.81 last 58.05) - open/close/unlink 11659 (time 1779310794.76 total 267.82 last 67.35) - open/close/unlink 12175 (time 1779310804.77 total 277.83 last 51.57) - open/close/unlink 12617 (time 1779310814.77 total 287.83 last 44.19) - open/close/unlink 13154 (time 1779310824.77 total 297.83 last 53.68) - open/close/unlink 13727 (time 1779310834.79 total 307.84 last 57.22) - open/close/unlink 14313 (time 1779310844.79 total 317.85 last 58.56) - open/close/unlink 14845 (time 1779310854.80 total 327.86 last 53.17) - open/close/unlink 15381 (time 1779310864.81 total 337.87 last 53.53) - open/close/unlink 15963 (time 1779310874.82 total 347.88 last 58.16) - open/close/unlink 16493 (time 1779310884.84 total 357.89 last 52.91) - open/close/unlink 16983 (time 1779310894.86 total 367.91 last 48.90) - open/close/unlink 17483 (time 1779310904.87 total 377.92 last 49.95) - open/close/unlink 17931 (time 1779310914.88 total 387.93 last 44.77) - open/close/unlink 18486 (time 1779310924.88 total 397.94 last 55.46) - open/close/unlink 18994 (time 1779310934.89 total 407.95 last 50.77) - open/close/unlink 19546 (time 1779310944.90 total 417.95 last 55.16) - open/close/unlink 20000 (time 1779310952.60 total 425.65 last 58.95) - open/close/unlink 20576 (time 1779310962.61 total 435.66 last 57.55) - open/close/unlink 21145 (time 1779310972.62 total 445.68 last 56.81) - open/close/unlink 21702 (time 1779310982.63 total 455.68 last 55.67) - open/close/unlink 22272 (time 1779310992.63 total 465.69 last 56.97) - open/close/unlink 22792 (time 1779311002.64 total 475.69 last 51.97) - open/close/unlink 23312 (time 1779311012.65 total 485.70 last 51.95) - open/close/unlink 23824 (time 1779311022.66 total 495.72 last 51.12) - open/close/unlink 24345 (time 1779311032.68 total 505.73 last 52.03) - open/close/unlink 24849 (time 1779311042.69 total 515.75 last 50.32) - open/close/unlink 25400 (time 1779311052.70 total 525.75 last 55.07) - open/close/unlink 25967 (time 1779311062.70 total 535.76 last 56.69) - open/close/unlink 26653 (time 1779311072.71 total 545.77 last 68.52) - open/close/unlink 27389 (time 1779311082.72 total 555.78 last 73.53) - open/close/unlink 28101 (time 1779311092.73 total 565.78 last 71.17) - open/close/unlink 28809 - open/close/unlink 498 (time 1779310537.40 total 10.02 last 49.69) - open/close/unlink 969 (time 1779310547.42 total 20.04 last 46.99) - open/close/unlink 1415 (time 1779310557.44 total 30.06 last 44.54) - open/close/unlink 1893 (time 1779310567.46 total 40.08 last 47.71) - open/close/unlink 2302 (time 1779310577.48 total 50.10 last 40.79) - open/close/unlink 2671 (time 1779310587.48 total 60.10 last 36.90) - open/close/unlink 3045 (time 1779310597.48 total 70.11 last 37.39) - open/close/unlink 3480 (time 1779310607.50 total 80.12 last 43.42) - open/close/unlink 3875 (time 1779310617.51 total 90.13 last 39.46) - open/close/unlink 4282 (time 1779310627.54 total 100.16 last 40.59) - open/close/unlink 4740 (time 1779310637.56 total 110.19 last 45.68) - open/close/unlink 5153 (time 1779310647.61 total 120.23 last 41.12) - open/close/unlink 5547 (time 1779310657.63 total 130.25 last 39.32) - open/close/unlink 5914 (time 1779310667.65 total 140.27 last 36.64) - open/close/unlink 6274 (time 1779310677.65 total 150.27 last 35.98) - open/close/unlink 6619 (time 1779310687.66 total 160.28 last 34.46) - open/close/unlink 7041 (time 1779310697.67 total 170.29 last 42.17) - open/close/unlink 7479 (time 1779310707.69 total 180.31 last 43.73) - open/close/unlink 7911 (time 1779310717.69 total 190.31 last 43.17) - open/close/unlink 8358 (time 1779310727.70 total 200.32 last 44.65) - open/close/unlink 8792 (time 1779310737.72 total 210.34 last 43.34) - open/close/unlink 9313 (time 1779310747.73 total 220.35 last 52.03) - open/close/unlink 9725 (time 1779310757.75 total 230.37 last 41.12) - open/close/unlink 10000 (time 1779310764.93 total 237.55 last 38.30) - open/close/unlink 10412 (time 1779310774.94 total 247.56 last 41.16) - open/close/unlink 10985 (time 1779310784.95 total 257.57 last 57.26) - open/close/unlink 11662 (time 1779310794.95 total 267.57 last 67.68) - open/close/unlink 12179 (time 1779310804.97 total 277.59 last 51.63) - open/close/unlink 12623 (time 1779310814.97 total 287.59 last 44.39) - open/close/unlink 13162 (time 1779310824.97 total 297.60 last 53.86) - open/close/unlink 13739 (time 1779310834.98 total 307.60 last 57.68) - open/close/unlink 14332 (time 1779310844.99 total 317.61 last 59.24) - open/close/unlink 14853 (time 1779310854.99 total 327.62 last 52.07) - open/close/unlink 15386 (time 1779310865.01 total 337.63 last 53.24) - open/close/unlink 15973 (time 1779310875.02 total 347.64 last 58.64) - open/close/unlink 16505 (time 1779310885.03 total 357.65 last 53.13) - open/close/unlink 17002 (time 1779310895.05 total 367.68 last 49.58) - open/close/unlink 17497 (time 1779310905.07 total 377.69 last 49.41) - open/close/unlink 17956 (time 1779310915.09 total 387.71 last 45.81) - open/close/unlink 18509 (time 1779310925.10 total 397.72 last 55.28) - open/close/unlink 19007 (time 1779310935.11 total 407.73 last 49.72) - open/close/unlink 19553 (time 1779310945.12 total 417.74 last 54.55) - open/close/unlink 20000 (time 1779310952.66 total 425.28 last 59.28) - open/close/unlink 20574 (time 1779310962.67 total 435.29 last 57.36) - open/close/unlink 21144 (time 1779310972.69 total 445.31 last 56.89) - open/close/unlink 21700 (time 1779310982.71 total 455.33 last 55.47) - open/close/unlink 22272 (time 1779310992.73 total 465.35 last 57.10) - open/close/unlink 22790 (time 1779311002.73 total 475.35 last 51.78) - open/close/unlink 23314 (time 1779311012.74 total 485.36 last 52.35) - open/close/unlink 23818 (time 1779311022.76 total 495.38 last 50.30) - open/close/unlink 24336 (time 1779311032.77 total 505.39 last 51.78) - open/close/unlink 24850 (time 1779311042.77 total 515.39 last 51.38) - open/close/unlink 25403 (time 1779311052.78 total 525.40 last 55.26) - open/close/unlink 25970 (time 1779311062.78 total 535.41 last 56.66) - open/close/unlink 26667 (time 1779311072.79 total 545.41 last 69.65) - open/close/unlink 27414 (time 1779311082.80 total 555.42 last 74.64) - open/close/unlink 28124 (time 1779311092.81 total 565.43 last 70.95) - open/close/unlink 28823 - open/close/unlink 498 (time 1779310537.38 total 10.01 last 49.77) - open/close/unlink 973 (time 1779310547.39 total 20.02 last 47.45) - open/close/unlink 1424 (time 1779310557.41 total 30.04 last 44.99) - open/close/unlink 1905 (time 1779310567.42 total 40.05 last 48.05) - open/close/unlink 2314 (time 1779310577.42 total 50.05 last 40.89) - open/close/unlink 2684 (time 1779310587.43 total 60.06 last 36.99) - open/close/unlink 3053 (time 1779310597.44 total 70.07 last 36.85) - open/close/unlink 3487 (time 1779310607.46 total 80.09 last 43.30) - open/close/unlink 3883 (time 1779310617.47 total 90.10 last 39.57) - open/close/unlink 4296 (time 1779310627.48 total 100.11 last 41.26) - open/close/unlink 4749 (time 1779310637.50 total 110.13 last 45.23) - open/close/unlink 5155 (time 1779310647.53 total 120.16 last 40.49) - open/close/unlink 5553 (time 1779310657.53 total 130.16 last 39.76) - open/close/unlink 5921 (time 1779310667.54 total 140.17 last 36.78) - open/close/unlink 6279 (time 1779310677.59 total 150.22 last 35.62) - open/close/unlink 6621 (time 1779310687.61 total 160.24 last 34.12) - open/close/unlink 7044 (time 1779310697.62 total 170.25 last 42.27) - open/close/unlink 7477 (time 1779310707.63 total 180.26 last 43.28) - open/close/unlink 7903 (time 1779310717.66 total 190.29 last 42.44) - open/close/unlink 8356 (time 1779310727.68 total 200.31 last 45.22) - open/close/unlink 8794 (time 1779310737.68 total 210.31 last 43.79) - open/close/unlink 9308 (time 1779310747.69 total 220.32 last 51.38) - open/close/unlink 9713 (time 1779310757.69 total 230.32 last 40.50) - open/close/unlink 10000 (time 1779310765.26 total 237.89 last 37.89) - open/close/unlink 10419 (time 1779310775.27 total 247.90 last 41.85) - open/close/unlink 11001 (time 1779310785.28 total 257.91 last 58.18) - open/close/unlink 11673 (time 1779310795.28 total 267.91 last 67.18) - open/close/unlink 12185 (time 1779310805.30 total 277.93 last 51.11) - open/close/unlink 12626 (time 1779310815.30 total 287.93 last 44.09) - open/close/unlink 13165 (time 1779310825.31 total 297.94 last 53.86) - open/close/unlink 13738 (time 1779310835.32 total 307.95 last 57.24) - open/close/unlink 14328 (time 1779310845.32 total 317.95 last 58.98) - open/close/unlink 14851 (time 1779310855.34 total 327.97 last 52.21) - open/close/unlink 15377 (time 1779310865.35 total 337.98 last 52.54) - open/close/unlink 15967 (time 1779310875.37 total 348.00 last 58.91) - open/close/unlink 16497 (time 1779310885.37 total 358.00 last 52.95) - open/close/unlink 16982 (time 1779310895.39 total 368.02 last 48.44) - open/close/unlink 17477 (time 1779310905.39 total 378.02 last 49.49) - open/close/unlink 17931 (time 1779310915.40 total 388.03 last 45.35) - open/close/unlink 18493 (time 1779310925.40 total 398.03 last 56.20) - open/close/unlink 18988 (time 1779310935.41 total 408.04 last 49.44) - open/close/unlink 19541 (time 1779310945.41 total 418.04 last 55.30) - open/close/unlink 20000 (time 1779310953.12 total 425.75 last 59.54) - open/close/unlink 20575 (time 1779310963.12 total 435.75 last 57.49) - open/close/unlink 21139 (time 1779310973.13 total 445.76 last 56.38) - open/close/unlink 21692 (time 1779310983.13 total 455.76 last 55.26) - open/close/unlink 22266 (time 1779310993.14 total 465.77 last 57.35) - open/close/unlink 22788 (time 1779311003.15 total 475.78 last 52.18) - open/close/unlink 23307 (time 1779311013.16 total 485.79 last 51.84) - open/close/unlink 23810 (time 1779311023.17 total 495.80 last 50.22) - open/close/unlink 24328 (time 1779311033.19 total 505.82 last 51.74) - open/close/unlink 24826 (time 1779311043.19 total 515.82 last 49.79) - open/close/unlink 25382 (time 1779311053.20 total 525.83 last 55.52) - open/close/unlink 25953 (time 1779311063.21 total 535.84 last 57.02) - open/close/unlink 26655 (time 1779311073.23 total 545.86 last 70.11) - open/close/unlink 27388 (time 1779311083.23 total 555.86 last 73.27) - open/close/unlink 28090 (time 1779311093.23 total 565.86 last 70.18) - open/close/unlink 28805 - open/close/unlink 501 (time 1779310537.40 total 10.04 last 49.89) - open/close/unlink 969 (time 1779310547.42 total 20.07 last 46.68) - open/close/unlink 1419 (time 1779310557.44 total 30.08 last 44.94) - open/close/unlink 1891 (time 1779310567.46 total 40.10 last 47.10) - open/close/unlink 2297 (time 1779310577.49 total 50.13 last 40.48) - open/close/unlink 2659 (time 1779310587.53 total 60.17 last 36.04) - open/close/unlink 3033 (time 1779310597.54 total 70.18 last 37.37) - open/close/unlink 3469 (time 1779310607.54 total 80.18 last 43.60) - open/close/unlink 3868 (time 1779310617.54 total 90.19 last 39.88) - open/close/unlink 4278 (time 1779310627.56 total 100.20 last 40.93) - open/close/unlink 4735 (time 1779310637.58 total 110.23 last 45.60) - open/close/unlink 5145 (time 1779310647.59 total 120.23 last 40.98) - open/close/unlink 5549 (time 1779310657.59 total 130.23 last 40.39) - open/close/unlink 5914 (time 1779310667.59 total 140.23 last 36.50) - open/close/unlink 6268 (time 1779310677.61 total 150.26 last 35.32) - open/close/unlink 6614 (time 1779310687.62 total 160.26 last 34.59) - open/close/unlink 7044 (time 1779310697.63 total 170.28 last 42.94) - open/close/unlink 7484 (time 1779310707.64 total 180.29 last 43.95) - open/close/unlink 7911 (time 1779310717.65 total 190.30 last 42.65) - open/close/unlink 8360 (time 1779310727.68 total 200.32 last 44.80) - open/close/unlink 8796 (time 1779310737.69 total 210.33 last 43.56) - open/close/unlink 9312 (time 1779310747.69 total 220.33 last 51.59) - open/close/unlink 9724 (time 1779310757.71 total 230.36 last 41.09) - open/close/unlink 10000 (time 1779310765.13 total 237.78 last 37.19) - open/close/unlink 10409 (time 1779310775.15 total 247.79 last 40.85) - open/close/unlink 10982 (time 1779310785.16 total 257.80 last 57.25) - open/close/unlink 11647 (time 1779310795.16 total 267.81 last 66.46) - open/close/unlink 12165 (time 1779310805.17 total 277.82 last 51.74) - open/close/unlink 12611 (time 1779310815.20 total 287.84 last 44.48) - open/close/unlink 13155 (time 1779310825.21 total 297.86 last 54.32) - open/close/unlink 13725 (time 1779310835.22 total 307.86 last 56.97) - open/close/unlink 14316 (time 1779310845.23 total 317.88 last 59.02) - open/close/unlink 14839 (time 1779310855.26 total 327.90 last 52.16) - open/close/unlink 15370 (time 1779310865.26 total 337.90 last 53.10) - open/close/unlink 15962 (time 1779310875.28 total 347.92 last 59.09) - open/close/unlink 16491 (time 1779310885.30 total 357.94 last 52.79) - open/close/unlink 16982 (time 1779310895.31 total 367.96 last 49.03) - open/close/unlink 17474 (time 1779310905.31 total 377.96 last 49.19) - open/close/unlink 17933 (time 1779310915.33 total 387.98 last 45.80) - open/close/unlink 18497 (time 1779310925.35 total 397.99 last 56.32) - open/close/unlink 18991 (time 1779310935.36 total 408.00 last 49.36) - open/close/unlink 19536 (time 1779310945.36 total 418.00 last 54.48) - open/close/unlink 20000 (time 1779310953.21 total 425.85 last 59.12) - open/close/unlink 20576 (time 1779310963.21 total 435.85 last 57.60) - open/close/unlink 21149 (time 1779310973.22 total 445.86 last 57.22) - open/close/unlink 21698 (time 1779310983.23 total 455.87 last 54.87) - open/close/unlink 22273 (time 1779310993.23 total 465.87 last 57.50) - open/close/unlink 22795 (time 1779311003.25 total 475.89 last 52.09) - open/close/unlink 23319 (time 1779311013.25 total 485.89 last 52.39) - open/close/unlink 23824 (time 1779311023.26 total 495.91 last 50.44) - open/close/unlink 24347 (time 1779311033.27 total 505.91 last 52.27) - open/close/unlink 24858 (time 1779311043.27 total 515.92 last 51.08) - open/close/unlink 25419 (time 1779311053.28 total 525.92 last 56.07) - open/close/unlink 25981 (time 1779311063.29 total 535.94 last 56.12) - open/close/unlink 26685 (time 1779311073.30 total 545.94 last 70.36) - open/close/unlink 27425 (time 1779311083.30 total 555.94 last 73.99) - open/close/unlink 28120 (time 1779311093.31 total 565.95 last 69.45) - open/close/unlink 28830 (time 1779311102.06 total 574.74 last 70.52) - open/close/unlink 29537 (time 1779311112.07 total 584.74 last 71.26) - open/close/unlink 30000 (time 1779311118.03 total 590.71 last 77.63) - open/close/unlink 30705 (time 1779311128.03 total 600.71 last 70.48) - open/close/unlink 31516 (time 1779311138.04 total 610.72 last 81.03) - open/close/unlink 32150 (time 1779311148.05 total 620.72 last 63.37) - open/close/unlink 32807 (time 1779311158.07 total 630.74 last 65.57) - open/close/unlink 33449 (time 1779311168.08 total 640.76 last 64.09) - open/close/unlink 34022 (time 1779311178.09 total 650.76 last 57.29) - open/close/unlink 34734 (time 1779311188.11 total 660.78 last 71.04) - open/close/unlink 35371 (time 1779311198.12 total 670.79 last 63.64) - open/close/unlink 35918 (time 1779311208.13 total 680.80 last 54.64) - open/close/unlink 36659 (time 1779311218.13 total 690.81 last 74.05) - open/close/unlink 37295 (time 1779311228.15 total 700.83 last 63.48) - open/close/unlink 37959 (time 1779311238.16 total 710.84 last 66.33) - open/close/unlink 38739 (time 1779311248.17 total 720.85 last 77.93) - open/close/unlink 39598 (time 1779311258.17 total 730.85 last 85.88) - open/close/unlink 40000 (time 1779311262.69 total 735.37 last 88.99) - open/close/unlink 40884 (time 1779311272.69 total 745.37 last 88.39) - open/close/unlink 41900 (time 1779311282.70 total 755.37 last 101.57) - open/close/unlink 42928 (time 1779311292.70 total 765.38 last 102.75) - open/close/unlink 43967 (time 1779311302.70 total 775.38 last 103.89) - open/close/unlink 44948 (time 1779311312.70 total 785.38 last 98.08) - open/close/unlink 45968 (time 1779311322.71 total 795.39 last 101.87) - open/close/unlink 47047 (time 1779311332.72 total 805.39 last 107.85) - open/close/unlink 47867 (time 1779311342.72 total 815.40 last 81.96) - open/close/unlink 48810 (time 1779311352.73 total 825.40 last 94.28) - open/close/unlink 49847 (time 1779311362.73 total 835.41 last 103.64) - open/close/unlink 50781 (time 1779311372.74 total 845.42 last 93.32) - open/close/unlink 51498 (time 1779311382.74 total 855.42 last 71.69) - open/close/unlink 52238 (time 1779311392.75 total 865.43 last 73.91) - open/close/unlink 52832 (time 1779311402.77 total 875.45 last 59.28) - open/close/unlink 53503 (time 1779311412.78 total 885.46 last 67.04) - open/close/unlink 54092 (time 1779311422.80 total 895.48 last 58.80) - open/close/unlink 54636 (time 1779311432.82 total 905.49 last 54.31) - open/close/unlink 55198 (time 1779311442.82 total 915.50 last 56.18) - open/close/unlink 55810 (time 1779311452.83 total 925.51 last 61.16) - open/close/unlink 56614 (time 1779311462.83 total 935.51 last 80.36) - open/close/unlink 57260 (time 1779311472.84 total 945.51 last 64.57) - open/close/unlink 58136 (time 1779311482.84 total 955.51 last 87.59) - open/close/unlink 58910 (time 1779311492.84 total 965.52 last 77.38) - open/close/unlink 59555 (time 1779311502.84 total 975.52 last 64.49) - open/close/unlink 60000 (time 1779311509.17 total 981.84 last 70.38) - open/close/unlink 60767 (time 1779311519.17 total 991.84 last 76.70) - open/close/unlink 61594 (time 1779311529.18 total 1001.85 last 82.63) - open/close/unlink 62312 (time 1779311539.18 total 1011.86 last 71.77) - open/close/unlink 63161 (time 1779311549.19 total 1021.86 last 84.85) - open/close/unlink 64065 (time 1779311559.19 total 1031.86 last 90.38) total: 64768 open/close/unlink in 1040.44 seconds: 62.25 ops/second (time 1779311102.81 total 575.43 last 69.86) - open/close/unlink 29524 (time 1779311112.81 total 585.44 last 70.08) - open/close/unlink 30000 (time 1779311118.77 total 591.39 last 79.93) - open/close/unlink 30700 (time 1779311128.77 total 601.39 last 69.98) - open/close/unlink 31495 (time 1779311138.77 total 611.40 last 79.49) - open/close/unlink 32125 (time 1779311148.80 total 621.42 last 62.87) - open/close/unlink 32790 (time 1779311158.82 total 631.44 last 66.33) - open/close/unlink 33421 (time 1779311168.86 total 641.48 last 62.87) - open/close/unlink 34040 (time 1779311178.86 total 651.48 last 61.86) - open/close/unlink 34725 (time 1779311188.87 total 661.49 last 68.47) - open/close/unlink 35346 (time 1779311198.88 total 671.50 last 62.03) - open/close/unlink 35887 (time 1779311208.88 total 681.50 last 54.10) - open/close/unlink 36631 (time 1779311218.88 total 691.50 last 74.37) - open/close/unlink 37261 (time 1779311228.89 total 701.51 last 62.93) - open/close/unlink 37924 (time 1779311238.91 total 711.53 last 66.20) - open/close/unlink 38731 (time 1779311248.92 total 721.54 last 80.62) - open/close/unlink 39611 (time 1779311258.93 total 731.55 last 87.92) - open/close/unlink 40000 (time 1779311263.49 total 736.11 last 85.24) - open/close/unlink 40897 (time 1779311273.50 total 746.12 last 89.65) - open/close/unlink 41915 (time 1779311283.50 total 756.12 last 101.75) - open/close/unlink 42931 (time 1779311293.50 total 766.12 last 101.59) - open/close/unlink 43975 (time 1779311303.51 total 776.13 last 104.32) - open/close/unlink 44955 (time 1779311313.51 total 786.13 last 97.97) - open/close/unlink 45958 (time 1779311323.52 total 796.14 last 100.24) - open/close/unlink 47041 (time 1779311333.52 total 806.14 last 108.30) - open/close/unlink 47858 (time 1779311343.52 total 816.15 last 81.65) - open/close/unlink 48813 (time 1779311353.52 total 826.15 last 95.50) - open/close/unlink 49850 (time 1779311363.53 total 836.15 last 103.64) - open/close/unlink 50742 (time 1779311373.53 total 846.15 last 89.19) - open/close/unlink 51460 (time 1779311383.53 total 856.15 last 71.79) - open/close/unlink 52178 (time 1779311393.54 total 866.17 last 71.71) - open/close/unlink 52768 (time 1779311403.56 total 876.18 last 58.91) - open/close/unlink 53430 (time 1779311413.58 total 886.20 last 66.09) - open/close/unlink 54019 (time 1779311423.59 total 896.21 last 58.84) - open/close/unlink 54557 (time 1779311433.60 total 906.22 last 53.73) - open/close/unlink 55135 (time 1779311443.66 total 916.29 last 57.44) - open/close/unlink 55775 (time 1779311453.67 total 926.29 last 63.98) - open/close/unlink 56565 (time 1779311463.68 total 936.30 last 78.92) - open/close/unlink 57212 (time 1779311473.68 total 946.30 last 64.69) - open/close/unlink 58092 (time 1779311483.69 total 956.31 last 87.89) - open/close/unlink 58857 (time 1779311493.70 total 966.32 last 76.45) - open/close/unlink 59498 (time 1779311503.71 total 976.33 last 64.03) - open/close/unlink 60000 (time 1779311510.78 total 983.40 last 71.04) - open/close/unlink 60758 (time 1779311520.78 total 993.40 last 75.74) - open/close/unlink 61594 (time 1779311530.78 total 1003.41 last 83.59) - open/close/unlink 62333 (time 1779311540.79 total 1013.41 last 73.84) - open/close/unlink 63192 (time 1779311550.81 total 1023.43 last 85.74) - open/close/unlink 64108 (time 1779311560.82 total 1033.44 last 91.55) - open/close/unlink 64760 (time 1779311570.85 total 1043.47 last 65.01) total: 64768 open/close/unlink in 1043.69 seconds: 62.06 ops/second (time 1779311103.31 total 575.95 last 70.98) - open/close/unlink 29537 (time 1779311113.31 total 585.96 last 70.67) - open/close/unlink 30000 (time 1779311119.07 total 591.71 last 80.45) - open/close/unlink 30709 (time 1779311129.07 total 601.72 last 70.87) - open/close/unlink 31503 (time 1779311139.08 total 611.72 last 79.38) - open/close/unlink 32128 (time 1779311149.08 total 621.73 last 62.45) - open/close/unlink 32802 (time 1779311159.09 total 631.73 last 67.36) - open/close/unlink 33422 (time 1779311169.12 total 641.76 last 61.81) - open/close/unlink 34054 (time 1779311179.13 total 651.77 last 63.15) - open/close/unlink 34738 (time 1779311189.13 total 661.78 last 68.38) - open/close/unlink 35356 (time 1779311199.13 total 671.78 last 61.79) - open/close/unlink 35913 (time 1779311209.15 total 681.79 last 55.62) - open/close/unlink 36664 (time 1779311219.16 total 691.80 last 75.04) - open/close/unlink 37290 (time 1779311229.16 total 701.80 last 62.57) - open/close/unlink 37963 (time 1779311239.17 total 711.82 last 67.23) - open/close/unlink 38765 (time 1779311249.17 total 721.82 last 80.19) - open/close/unlink 39654 (time 1779311259.18 total 731.82 last 88.86) - open/close/unlink 40000 (time 1779311263.29 total 735.93 last 84.15) - open/close/unlink 40902 (time 1779311273.30 total 745.94 last 90.11) - open/close/unlink 41919 (time 1779311283.30 total 755.95 last 101.64) - open/close/unlink 42947 (time 1779311293.31 total 765.95 last 102.74) - open/close/unlink 43976 (time 1779311303.31 total 775.96 last 102.85) - open/close/unlink 44955 (time 1779311313.32 total 785.96 last 97.87) - open/close/unlink 45951 (time 1779311323.32 total 795.97 last 99.53) - open/close/unlink 47027 (time 1779311333.33 total 805.98 last 107.53) - open/close/unlink 47833 (time 1779311343.34 total 815.98 last 80.55) - open/close/unlink 48783 (time 1779311353.34 total 825.99 last 94.94) - open/close/unlink 49807 (time 1779311363.35 total 835.99 last 102.34) - open/close/unlink 50701 (time 1779311373.36 total 846.01 last 89.28) - open/close/unlink 51435 (time 1779311383.37 total 856.01 last 73.35) - open/close/unlink 52162 (time 1779311393.37 total 866.02 last 72.68) - open/close/unlink 52746 (time 1779311403.38 total 876.03 last 58.34) - open/close/unlink 53403 (time 1779311413.39 total 886.03 last 65.66) - open/close/unlink 53997 (time 1779311423.40 total 896.04 last 59.35) - open/close/unlink 54528 (time 1779311433.41 total 906.05 last 53.06) - open/close/unlink 55106 (time 1779311443.42 total 916.06 last 57.74) - open/close/unlink 55729 (time 1779311453.42 total 926.07 last 62.26) - open/close/unlink 56513 (time 1779311463.43 total 936.07 last 78.35) - open/close/unlink 57164 (time 1779311473.44 total 946.08 last 65.03) - open/close/unlink 58054 (time 1779311483.44 total 956.09 last 88.98) - open/close/unlink 58820 (time 1779311493.45 total 966.09 last 76.57) - open/close/unlink 59461 (time 1779311503.46 total 976.10 last 64.02) - open/close/unlink 60000 (time 1779311511.07 total 983.71 last 70.84) - open/close/unlink 60750 (time 1779311521.08 total 993.72 last 74.91) - open/close/unlink 61594 (time 1779311531.09 total 1003.73 last 84.32) - open/close/unlink 62329 (time 1779311541.10 total 1013.74 last 73.44) - open/close/unlink 63184 (time 1779311551.14 total 1023.78 last 85.13) - open/close/unlink 64111 (time 1779311561.15 total 1033.79 last 92.62) - open/close/unlink 64748 (time 1779311571.17 total 1043.81 last 63.57) total: 64768 open/close/unlink in 1044.70 seconds: 62.00 ops/second (time 1779311102.73 total 575.79 last 70.78) - open/close/unlink 29511 (time 1779311112.73 total 585.79 last 70.17) - open/close/unlink 30000 (time 1779311118.83 total 591.88 last 80.24) - open/close/unlink 30701 (time 1779311128.83 total 601.89 last 70.06) - open/close/unlink 31490 (time 1779311138.84 total 611.89 last 78.88) - open/close/unlink 32120 (time 1779311148.85 total 621.91 last 62.88) - open/close/unlink 32780 (time 1779311158.87 total 631.93 last 65.90) - open/close/unlink 33407 (time 1779311168.89 total 641.95 last 62.55) - open/close/unlink 34034 (time 1779311178.89 total 651.95 last 62.69) - open/close/unlink 34721 (time 1779311188.90 total 661.95 last 68.68) - open/close/unlink 35342 (time 1779311198.91 total 671.96 last 62.04) - open/close/unlink 35888 (time 1779311208.91 total 681.97 last 54.60) - open/close/unlink 36628 (time 1779311218.92 total 691.98 last 73.92) - open/close/unlink 37260 (time 1779311228.96 total 702.02 last 62.94) - open/close/unlink 37925 (time 1779311238.96 total 712.02 last 66.49) - open/close/unlink 38733 (time 1779311248.97 total 722.03 last 80.72) - open/close/unlink 39613 (time 1779311258.98 total 732.03 last 87.95) - open/close/unlink 40000 (time 1779311263.55 total 736.60 last 84.67) - open/close/unlink 40906 (time 1779311273.55 total 746.61 last 90.56) - open/close/unlink 41931 (time 1779311283.56 total 756.62 last 102.43) - open/close/unlink 42952 (time 1779311293.56 total 766.62 last 102.08) - open/close/unlink 43990 (time 1779311303.57 total 776.62 last 103.75) - open/close/unlink 44965 (time 1779311313.57 total 786.63 last 97.44) - open/close/unlink 45971 (time 1779311323.58 total 796.63 last 100.56) - open/close/unlink 47043 (time 1779311333.59 total 806.65 last 107.06) - open/close/unlink 47859 (time 1779311343.59 total 816.65 last 81.57) - open/close/unlink 48806 (time 1779311353.59 total 826.65 last 94.70) - open/close/unlink 49835 (time 1779311363.60 total 836.66 last 102.83) - open/close/unlink 50721 (time 1779311373.61 total 846.66 last 88.54) - open/close/unlink 51440 (time 1779311383.61 total 856.66 last 71.89) - open/close/unlink 52163 (time 1779311393.62 total 866.68 last 72.18) - open/close/unlink 52754 (time 1779311403.63 total 876.68 last 59.09) - open/close/unlink 53413 (time 1779311413.64 total 886.70 last 65.78) - open/close/unlink 54008 (time 1779311423.66 total 896.72 last 59.39) - open/close/unlink 54529 (time 1779311433.67 total 906.73 last 52.05) - open/close/unlink 55108 (time 1779311443.68 total 916.74 last 57.85) - open/close/unlink 55750 (time 1779311453.69 total 926.74 last 64.16) - open/close/unlink 56531 (time 1779311463.70 total 936.75 last 78.03) - open/close/unlink 57182 (time 1779311473.71 total 946.76 last 65.04) - open/close/unlink 58067 (time 1779311483.71 total 956.77 last 88.44) - open/close/unlink 58826 (time 1779311493.72 total 966.78 last 75.84) - open/close/unlink 59471 (time 1779311503.72 total 976.78 last 64.48) - open/close/unlink 60000 (time 1779311511.16 total 984.22 last 71.15) - open/close/unlink 60752 (time 1779311521.17 total 994.22 last 75.13) - open/close/unlink 61583 (time 1779311531.18 total 1004.24 last 82.99) - open/close/unlink 62317 (time 1779311541.19 total 1014.25 last 73.33) - open/close/unlink 63169 (time 1779311551.20 total 1024.25 last 85.15) - open/close/unlink 64103 (time 1779311561.20 total 1034.25 last 93.39) - open/close/unlink 64734 (time 1779311571.21 total 1044.26 last 63.04) total: 64768 open/close/unlink in 1045.73 seconds: 61.94 ops/second (time 1779311103.24 total 575.87 last 71.47) - open/close/unlink 29517 (time 1779311113.24 total 585.87 last 71.16) - open/close/unlink 30000 (time 1779311119.22 total 591.85 last 80.81) - open/close/unlink 30702 (time 1779311129.22 total 601.85 last 70.20) - open/close/unlink 31493 (time 1779311139.22 total 611.85 last 79.08) - open/close/unlink 32115 (time 1779311149.23 total 621.86 last 62.17) - open/close/unlink 32794 (time 1779311159.23 total 631.86 last 67.86) - open/close/unlink 33397 (time 1779311169.25 total 641.88 last 60.22) - open/close/unlink 34027 (time 1779311179.25 total 651.89 last 62.94) - open/close/unlink 34711 (time 1779311189.26 total 661.89 last 68.40) - open/close/unlink 35330 (time 1779311199.28 total 671.91 last 61.76) - open/close/unlink 35882 (time 1779311209.29 total 681.92 last 55.15) - open/close/unlink 36624 (time 1779311219.29 total 691.92 last 74.16) - open/close/unlink 37245 (time 1779311229.29 total 701.92 last 62.09) - open/close/unlink 37921 (time 1779311239.31 total 711.94 last 67.47) - open/close/unlink 38720 (time 1779311249.33 total 721.96 last 79.80) - open/close/unlink 39610 (time 1779311259.33 total 731.96 last 88.94) - open/close/unlink 40000 (time 1779311263.86 total 736.50 last 86.04) - open/close/unlink 40891 (time 1779311273.87 total 746.50 last 89.08) - open/close/unlink 41918 (time 1779311283.87 total 756.50 last 102.68) - open/close/unlink 42936 (time 1779311293.87 total 766.50 last 101.75) - open/close/unlink 43975 (time 1779311303.88 total 776.51 last 103.87) - open/close/unlink 44941 (time 1779311313.88 total 786.51 last 96.56) - open/close/unlink 45937 (time 1779311323.88 total 796.51 last 99.58) - open/close/unlink 47018 (time 1779311333.89 total 806.52 last 108.01) - open/close/unlink 47831 (time 1779311343.89 total 816.52 last 81.29) - open/close/unlink 48790 (time 1779311353.90 total 826.53 last 95.87) - open/close/unlink 49814 (time 1779311363.90 total 836.53 last 102.36) - open/close/unlink 50682 (time 1779311373.91 total 846.54 last 86.74) - open/close/unlink 51415 (time 1779311383.91 total 856.54 last 73.28) - open/close/unlink 52132 (time 1779311393.92 total 866.55 last 71.64) - open/close/unlink 52717 (time 1779311403.93 total 876.56 last 58.42) - open/close/unlink 53374 (time 1779311413.94 total 886.57 last 65.64) - open/close/unlink 53961 (time 1779311423.95 total 896.58 last 58.65) - open/close/unlink 54488 (time 1779311433.95 total 906.58 last 52.69) - open/close/unlink 55064 (time 1779311443.96 total 916.59 last 57.54) - open/close/unlink 55704 (time 1779311453.98 total 926.61 last 63.91) - open/close/unlink 56496 (time 1779311463.99 total 936.62 last 79.10) - open/close/unlink 57157 (time 1779311474.00 total 946.63 last 66.05) - open/close/unlink 58043 (time 1779311484.01 total 956.64 last 88.51) - open/close/unlink 58796 (time 1779311494.01 total 966.64 last 75.26) - open/close/unlink 59435 (time 1779311504.01 total 976.64 last 63.88) - open/close/unlink 60000 (time 1779311511.95 total 984.58 last 71.16) - open/close/unlink 60764 (time 1779311521.96 total 994.59 last 76.35) - open/close/unlink 61581 (time 1779311531.97 total 1004.60 last 81.63) - open/close/unlink 62350 (time 1779311541.98 total 1014.61 last 76.84) - open/close/unlink 63195 (time 1779311551.98 total 1024.61 last 84.46) - open/close/unlink 64129 (time 1779311561.99 total 1034.62 last 93.32) - open/close/unlink 64721 (time 1779311572.03 total 1044.66 last 58.96) total: 64768 open/close/unlink in 1046.26 seconds: 61.90 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg409-server: cannot open 'lustre-ost2': no such pool pdsh@oleg409-client: oleg409-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg409-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg409-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg409-server stop mds service on oleg409-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg409-server unloading modules via unload_modules_local on: 'oleg409-server' oleg409-server: oleg409-server.virtnet: executing unload_modules_local oleg409-server: modules unloaded. pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-client: ssh exited with exit code 2 pdsh@oleg409-client: oleg409-server: ssh exited with exit code 2