== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:54:47 (1788360887) start mds service on oleg228-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg228-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg228-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg228-server: oleg228-server.virtnet: executing set_default_debug -1 all pdsh@oleg228-client: oleg228-server: ssh exited with exit code 1 Started lustre-OST0000 oleg228-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg228-client.virtnet: -o user_xattr,flock 192.168.202.128@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 413 (time 1788360935.84 total 10.00 last 41.29) - open/close/unlink 887 (time 1788360945.85 total 20.01 last 47.35) - open/close/unlink 1337 (time 1788360955.87 total 30.03 last 44.91) - open/close/unlink 1702 (time 1788360965.90 total 40.07 last 36.37) - open/close/unlink 2056 (time 1788360975.92 total 50.09 last 35.34) - open/close/unlink 2415 (time 1788360985.95 total 60.12 last 35.78) - open/close/unlink 2805 (time 1788360995.96 total 70.13 last 38.96) - open/close/unlink 3202 (time 1788361005.98 total 80.15 last 39.63) - open/close/unlink 3603 (time 1788361016.00 total 90.17 last 40.03) - open/close/unlink 3994 (time 1788361026.02 total 100.18 last 39.03) - open/close/unlink 4333 (time 1788361036.04 total 110.21 last 33.82) - open/close/unlink 4742 (time 1788361046.06 total 120.22 last 40.83) - open/close/unlink 5100 (time 1788361056.08 total 130.25 last 35.71) - open/close/unlink 5471 (time 1788361066.10 total 140.27 last 37.03) - open/close/unlink 5839 (time 1788361076.11 total 150.27 last 36.78) - open/close/unlink 6241 (time 1788361086.13 total 160.30 last 40.09) - open/close/unlink 6632 (time 1788361096.13 total 170.30 last 39.10) - open/close/unlink 7088 (time 1788361106.16 total 180.33 last 45.48) - open/close/unlink 7482 (time 1788361116.18 total 190.35 last 39.33) - open/close/unlink 7934 (time 1788361126.19 total 200.35 last 45.16) - open/close/unlink 8390 (time 1788361136.21 total 210.38 last 45.50) - open/close/unlink 8867 (time 1788361146.24 total 220.40 last 47.58) - open/close/unlink 9371 (time 1788361156.24 total 230.41 last 50.37) - open/close/unlink 9965 (time 1788361166.24 total 240.41 last 59.40) - open/close/unlink 10471 (time 1788361176.24 total 250.41 last 50.59) - open/close/unlink 10971 (time 1788361186.27 total 260.44 last 49.87) - open/close/unlink 11436 (time 1788361196.30 total 270.46 last 46.38) - open/close/unlink 11931 (time 1788361206.31 total 280.48 last 49.40) - open/close/unlink 12401 (time 1788361216.33 total 290.49 last 46.95) - open/close/unlink 12936 (time 1788361226.34 total 300.51 last 53.40) - open/close/unlink 13446 (time 1788361236.36 total 310.52 last 50.94) - open/close/unlink 13895 (time 1788361246.37 total 320.53 last 44.86) - open/close/unlink 14302 (time 1788361256.37 total 330.54 last 40.67) - open/close/unlink 14697 (time 1788361266.39 total 340.56 last 39.43) - open/close/unlink 15147 (time 1788361276.40 total 350.57 last 44.95) - open/close/unlink 15577 (time 1788361286.42 total 360.59 last 42.91) - open/close/unlink 15993 (time 1788361296.43 total 370.60 last 41.58) - open/close/unlink 16488 (time 1788361306.43 total 380.60 last 49.49) - open/close/unlink 17002 (time 1788361316.45 total 390.62 last 51.30) - open/close/unlink 17499 (time 1788361326.47 total 400.64 last 49.58) - open/close/unlink 17954 (time 1788361336.49 total 410.66 last 45.43) - open/close/unlink 18403 (time 1788361346.50 total 420.67 last 44.85) - open/close/unlink 18842 (time 1788361356.51 total 430.68 last 43.86) - open/close/unlink 19299 (time 1788361366.52 total 440.69 last 45.65) - open/close/unlink 19762 (time 1788361376.52 total 450.69 last 46.30) - open/close/unlink 20000 (time 1788361381.71 total 455.88 last 45.86) - open/close/unlink 20488 (time 1788361391.72 total 465.88 last 48.78) - open/close/unlink 21016 (time 1788361401.73 total 475.90 last 52.73) - open/close/unlink 21539 (time 1788361411.73 total 485.90 last 52.30) - open/close/unlink 21992 (time 1788361421.74 total 495.91 last 45.25) - open/close/unlink 22526 (time 1788361431.77 total 505.93 last 53.27) - open/close/unlink 23104 (time 1788361441.77 total 515.94 last 57.78) - open/close/unlink 23586 (time 1788361451.77 total 525.94 last 48.19) - open/close/unlink 24083 (time 1788361461.78 total 535.95 last 49.67) - open/close/unlink 24680 (time 1788361471.79 total 545.96 last 59.64) - open/close/unlink 25362 (time 1788361481.79 total 555.96 last 68.19) - open/close/unlink 26042 (time 1788361491.80 total 565.97 last 67.91) - open/close/unlink 26568 - open/close/unlink 409 (time 1788360936.10 total 10.02 last 40.84) - open/close/unlink 888 (time 1788360946.14 total 20.05 last 47.73) - open/close/unlink 1335 (time 1788360956.15 total 30.07 last 44.62) - open/close/unlink 1711 (time 1788360966.18 total 40.10 last 37.49) - open/close/unlink 2063 (time 1788360976.21 total 50.12 last 35.10) - open/close/unlink 2431 (time 1788360986.21 total 60.13 last 36.79) - open/close/unlink 2817 (time 1788360996.23 total 70.14 last 38.55) - open/close/unlink 3213 (time 1788361006.25 total 80.16 last 39.52) - open/close/unlink 3611 (time 1788361016.26 total 90.17 last 39.75) - open/close/unlink 3995 (time 1788361026.27 total 100.19 last 38.34) - open/close/unlink 4329 (time 1788361036.29 total 110.20 last 33.36) - open/close/unlink 4745 (time 1788361046.32 total 120.24 last 41.45) - open/close/unlink 5100 (time 1788361056.32 total 130.24 last 35.50) - open/close/unlink 5467 (time 1788361066.32 total 140.24 last 36.70) - open/close/unlink 5839 (time 1788361076.34 total 150.25 last 37.15) - open/close/unlink 6247 (time 1788361086.35 total 160.26 last 40.77) - open/close/unlink 6638 (time 1788361096.36 total 170.28 last 39.03) - open/close/unlink 7092 (time 1788361106.38 total 180.29 last 45.33) - open/close/unlink 7485 (time 1788361116.39 total 190.30 last 39.27) - open/close/unlink 7951 (time 1788361126.40 total 200.31 last 46.53) - open/close/unlink 8409 (time 1788361136.41 total 210.32 last 45.77) - open/close/unlink 8888 (time 1788361146.41 total 220.33 last 47.87) - open/close/unlink 9394 (time 1788361156.42 total 230.34 last 50.55) - open/close/unlink 9993 (time 1788361166.42 total 240.34 last 59.88) - open/close/unlink 10497 (time 1788361176.43 total 250.34 last 50.39) - open/close/unlink 11004 (time 1788361186.44 total 260.36 last 50.62) - open/close/unlink 11468 (time 1788361196.45 total 270.36 last 46.37) - open/close/unlink 11952 (time 1788361206.45 total 280.36 last 48.40) - open/close/unlink 12417 (time 1788361216.46 total 290.37 last 46.46) - open/close/unlink 12958 (time 1788361226.48 total 300.40 last 53.95) - open/close/unlink 13467 (time 1788361236.50 total 310.41 last 50.84) - open/close/unlink 13914 (time 1788361246.55 total 320.46 last 44.47) - open/close/unlink 14321 (time 1788361256.57 total 330.48 last 40.62) - open/close/unlink 14720 (time 1788361266.58 total 340.49 last 39.86) - open/close/unlink 15164 (time 1788361276.58 total 350.50 last 44.38) - open/close/unlink 15596 (time 1788361286.60 total 360.51 last 43.13) - open/close/unlink 16012 (time 1788361296.62 total 370.53 last 41.52) - open/close/unlink 16519 (time 1788361306.62 total 380.54 last 50.66) - open/close/unlink 17022 (time 1788361316.62 total 390.54 last 50.30) - open/close/unlink 17505 (time 1788361326.63 total 400.54 last 48.29) - open/close/unlink 17958 (time 1788361336.63 total 410.55 last 45.27) - open/close/unlink 18416 (time 1788361346.65 total 420.56 last 45.74) - open/close/unlink 18853 (time 1788361356.66 total 430.57 last 43.66) - open/close/unlink 19309 (time 1788361366.67 total 440.59 last 45.53) - open/close/unlink 19771 (time 1788361376.70 total 450.61 last 46.09) - open/close/unlink 20000 (time 1788361381.76 total 455.68 last 45.21) - open/close/unlink 20488 (time 1788361391.77 total 465.69 last 48.74) - open/close/unlink 21022 (time 1788361401.77 total 475.69 last 53.40) - open/close/unlink 21544 (time 1788361411.81 total 485.72 last 52.04) - open/close/unlink 22001 (time 1788361421.83 total 495.75 last 45.57) - open/close/unlink 22536 (time 1788361431.85 total 505.76 last 53.41) - open/close/unlink 23118 (time 1788361441.85 total 515.77 last 58.19) - open/close/unlink 23604 (time 1788361451.87 total 525.78 last 48.53) - open/close/unlink 24098 (time 1788361461.87 total 535.79 last 49.37) - open/close/unlink 24696 (time 1788361471.89 total 545.80 last 59.72) - open/close/unlink 25375 (time 1788361481.90 total 555.81 last 67.84) - open/close/unlink 26055 (time 1788361491.90 total 565.82 last 67.96) - open/close/unlink 26575 - open/close/unlink 411 (time 1788360936.13 total 10.02 last 41.02) - open/close/unlink 885 (time 1788360946.14 total 20.03 last 47.35) - open/close/unlink 1328 (time 1788360956.17 total 30.06 last 44.18) - open/close/unlink 1696 (time 1788360966.18 total 40.07 last 36.76) - open/close/unlink 2047 (time 1788360976.21 total 50.10 last 34.98) - open/close/unlink 2414 (time 1788360986.23 total 60.12 last 36.63) - open/close/unlink 2804 (time 1788360996.24 total 70.13 last 38.96) - open/close/unlink 3201 (time 1788361006.25 total 80.14 last 39.67) - open/close/unlink 3599 (time 1788361016.25 total 90.14 last 39.79) - open/close/unlink 3975 (time 1788361026.27 total 100.17 last 37.51) - open/close/unlink 4312 (time 1788361036.31 total 110.21 last 33.57) - open/close/unlink 4733 (time 1788361046.34 total 120.23 last 42.00) - open/close/unlink 5092 (time 1788361056.35 total 130.24 last 35.85) - open/close/unlink 5465 (time 1788361066.37 total 140.26 last 37.24) - open/close/unlink 5823 (time 1788361076.39 total 150.28 last 35.72) - open/close/unlink 6227 (time 1788361086.41 total 160.30 last 40.32) - open/close/unlink 6619 (time 1788361096.42 total 170.31 last 39.17) - open/close/unlink 7080 (time 1788361106.42 total 180.31 last 46.09) - open/close/unlink 7465 (time 1788361116.43 total 190.32 last 38.47) - open/close/unlink 7928 (time 1788361126.44 total 200.33 last 46.26) - open/close/unlink 8389 (time 1788361136.46 total 210.35 last 46.00) - open/close/unlink 8870 (time 1788361146.47 total 220.36 last 48.05) - open/close/unlink 9372 (time 1788361156.48 total 230.37 last 50.15) - open/close/unlink 9967 (time 1788361166.48 total 240.37 last 59.48) - open/close/unlink 10470 (time 1788361176.48 total 250.37 last 50.29) - open/close/unlink 10970 (time 1788361186.50 total 260.39 last 49.91) - open/close/unlink 11439 (time 1788361196.53 total 270.42 last 46.78) - open/close/unlink 11920 (time 1788361206.54 total 280.43 last 48.05) - open/close/unlink 12390 (time 1788361216.54 total 290.43 last 46.99) - open/close/unlink 12936 (time 1788361226.54 total 300.43 last 54.59) - open/close/unlink 13447 (time 1788361236.55 total 310.44 last 51.05) - open/close/unlink 13888 (time 1788361246.58 total 320.47 last 43.98) - open/close/unlink 14298 (time 1788361256.60 total 330.49 last 40.93) - open/close/unlink 14693 (time 1788361266.60 total 340.49 last 39.47) - open/close/unlink 15141 (time 1788361276.62 total 350.51 last 44.72) - open/close/unlink 15569 (time 1788361286.62 total 360.51 last 42.80) - open/close/unlink 15991 (time 1788361296.63 total 370.52 last 42.15) - open/close/unlink 16494 (time 1788361306.65 total 380.54 last 50.21) - open/close/unlink 16997 (time 1788361316.66 total 390.55 last 50.26) - open/close/unlink 17494 (time 1788361326.67 total 400.56 last 49.63) - open/close/unlink 17950 (time 1788361336.70 total 410.59 last 45.50) - open/close/unlink 18407 (time 1788361346.71 total 420.60 last 45.65) - open/close/unlink 18844 (time 1788361356.73 total 430.62 last 43.60) - open/close/unlink 19301 (time 1788361366.73 total 440.62 last 45.68) - open/close/unlink 19763 (time 1788361376.74 total 450.64 last 46.15) - open/close/unlink 20000 (time 1788361381.89 total 455.78 last 46.04) - open/close/unlink 20493 (time 1788361391.90 total 465.79 last 49.25) - open/close/unlink 21032 (time 1788361401.90 total 475.79 last 53.90) - open/close/unlink 21559 (time 1788361411.92 total 485.81 last 52.61) - open/close/unlink 22008 (time 1788361421.92 total 495.82 last 44.88) - open/close/unlink 22543 (time 1788361431.95 total 505.84 last 53.36) - open/close/unlink 23127 (time 1788361441.97 total 515.86 last 58.29) - open/close/unlink 23614 (time 1788361451.98 total 525.87 last 48.64) - open/close/unlink 24117 (time 1788361461.99 total 535.88 last 50.25) - open/close/unlink 24713 (time 1788361472.01 total 545.90 last 59.51) - open/close/unlink 25403 (time 1788361482.01 total 555.91 last 68.96) - open/close/unlink 26084 (time 1788361492.03 total 565.92 last 68.01) - open/close/unlink 26606 - open/close/unlink 414 (time 1788360936.12 total 10.01 last 41.37) - open/close/unlink 882 (time 1788360946.13 total 20.02 last 46.73) - open/close/unlink 1322 (time 1788360956.15 total 30.04 last 43.92) - open/close/unlink 1699 (time 1788360966.18 total 40.07 last 37.57) - open/close/unlink 2045 (time 1788360976.24 total 50.13 last 34.39) - open/close/unlink 2418 (time 1788360986.25 total 60.14 last 37.28) - open/close/unlink 2810 (time 1788360996.28 total 70.17 last 39.07) - open/close/unlink 3208 (time 1788361006.30 total 80.19 last 39.74) - open/close/unlink 3605 (time 1788361016.30 total 90.19 last 39.69) - open/close/unlink 3989 (time 1788361026.32 total 100.21 last 38.31) - open/close/unlink 4316 (time 1788361036.34 total 110.23 last 32.64) - open/close/unlink 4744 (time 1788361046.35 total 120.24 last 42.77) - open/close/unlink 5094 (time 1788361056.36 total 130.25 last 34.97) - open/close/unlink 5469 (time 1788361066.38 total 140.27 last 37.43) - open/close/unlink 5835 (time 1788361076.39 total 150.29 last 36.54) - open/close/unlink 6235 (time 1788361086.41 total 160.30 last 39.93) - open/close/unlink 6629 (time 1788361096.42 total 170.31 last 39.39) - open/close/unlink 7093 (time 1788361106.43 total 180.32 last 46.34) - open/close/unlink 7484 (time 1788361116.43 total 190.32 last 39.09) - open/close/unlink 7941 (time 1788361126.43 total 200.32 last 45.70) - open/close/unlink 8397 (time 1788361136.44 total 210.33 last 45.55) - open/close/unlink 8875 (time 1788361146.45 total 220.34 last 47.76) - open/close/unlink 9377 (time 1788361156.48 total 230.37 last 50.08) - open/close/unlink 9980 (time 1788361166.50 total 240.39 last 60.17) - open/close/unlink 10479 (time 1788361176.50 total 250.39 last 49.89) - open/close/unlink 10983 (time 1788361186.50 total 260.39 last 50.38) - open/close/unlink 11442 (time 1788361196.51 total 270.40 last 45.86) - open/close/unlink 11930 (time 1788361206.53 total 280.42 last 48.72) - open/close/unlink 12398 (time 1788361216.55 total 290.44 last 46.70) - open/close/unlink 12936 (time 1788361226.56 total 300.45 last 53.76) - open/close/unlink 13447 (time 1788361236.57 total 310.46 last 51.04) - open/close/unlink 13890 (time 1788361246.58 total 320.47 last 44.27) - open/close/unlink 14298 (time 1788361256.58 total 330.47 last 40.78) - open/close/unlink 14700 (time 1788361266.59 total 340.49 last 40.15) - open/close/unlink 15149 (time 1788361276.62 total 350.51 last 44.79) - open/close/unlink 15577 (time 1788361286.62 total 360.51 last 42.79) - open/close/unlink 16001 (time 1788361296.64 total 370.53 last 42.31) - open/close/unlink 16506 (time 1788361306.66 total 380.55 last 50.43) - open/close/unlink 17010 (time 1788361316.67 total 390.56 last 50.31) - open/close/unlink 17497 (time 1788361326.68 total 400.57 last 48.67) - open/close/unlink 17949 (time 1788361336.70 total 410.59 last 45.11) - open/close/unlink 18400 (time 1788361346.72 total 420.61 last 45.02) - open/close/unlink 18847 (time 1788361356.74 total 430.63 last 44.59) - open/close/unlink 19302 (time 1788361366.75 total 440.64 last 45.48) - open/close/unlink 19757 (time 1788361376.75 total 450.64 last 45.47) - open/close/unlink 20000 (time 1788361381.97 total 455.86 last 46.61) - open/close/unlink 20484 (time 1788361391.98 total 465.87 last 48.35) - open/close/unlink 21017 (time 1788361401.99 total 475.88 last 53.23) - open/close/unlink 21537 (time 1788361412.00 total 485.89 last 51.94) - open/close/unlink 21992 (time 1788361422.04 total 495.93 last 45.33) - open/close/unlink 22525 (time 1788361432.06 total 505.95 last 53.18) - open/close/unlink 23108 (time 1788361442.08 total 515.97 last 58.20) - open/close/unlink 23592 (time 1788361452.08 total 525.97 last 48.39) - open/close/unlink 24094 (time 1788361462.08 total 535.97 last 50.19) - open/close/unlink 24687 (time 1788361472.09 total 545.98 last 59.28) - open/close/unlink 25377 (time 1788361482.10 total 556.00 last 68.87) - open/close/unlink 26055 (time 1788361492.12 total 566.01 last 67.70) - open/close/unlink 26572 - open/close/unlink 419 (time 1788360935.57 total 10.01 last 41.88) - open/close/unlink 895 (time 1788360945.58 total 20.02 last 47.55) - open/close/unlink 1343 (time 1788360955.60 total 30.04 last 44.71) - open/close/unlink 1702 (time 1788360965.61 total 40.05 last 35.86) - open/close/unlink 2056 (time 1788360975.65 total 50.08 last 35.29) - open/close/unlink 2418 (time 1788360985.67 total 60.10 last 36.12) - open/close/unlink 2812 (time 1788360995.68 total 70.11 last 39.37) - open/close/unlink 3207 (time 1788361005.68 total 80.11 last 39.48) - open/close/unlink 3607 (time 1788361015.68 total 90.12 last 39.98) - open/close/unlink 3993 (time 1788361025.70 total 100.13 last 38.54) - open/close/unlink 4327 (time 1788361035.73 total 110.16 last 33.30) - open/close/unlink 4744 (time 1788361045.73 total 120.16 last 41.69) - open/close/unlink 5100 (time 1788361055.77 total 130.20 last 35.47) - open/close/unlink 5462 (time 1788361065.78 total 140.22 last 36.14) - open/close/unlink 5828 (time 1788361075.79 total 150.22 last 36.58) - open/close/unlink 6232 (time 1788361085.80 total 160.23 last 40.36) - open/close/unlink 6622 (time 1788361095.80 total 170.24 last 38.99) - open/close/unlink 7081 (time 1788361105.80 total 180.24 last 45.89) - open/close/unlink 7465 (time 1788361115.82 total 190.25 last 38.36) - open/close/unlink 7922 (time 1788361125.83 total 200.26 last 45.63) - open/close/unlink 8379 (time 1788361135.84 total 210.27 last 45.67) - open/close/unlink 8857 (time 1788361145.85 total 220.28 last 47.77) - open/close/unlink 9353 (time 1788361155.86 total 230.29 last 49.53) - open/close/unlink 9930 (time 1788361165.87 total 240.30 last 57.65) - open/close/unlink 10440 (time 1788361175.88 total 250.32 last 50.92) - open/close/unlink 10953 (time 1788361185.91 total 260.34 last 51.18) - open/close/unlink 11411 (time 1788361195.92 total 270.35 last 45.75) - open/close/unlink 11893 (time 1788361205.92 total 280.35 last 48.19) - open/close/unlink 12355 (time 1788361215.92 total 290.36 last 46.19) - open/close/unlink 12886 (time 1788361225.93 total 300.36 last 53.06) - open/close/unlink 13398 (time 1788361235.94 total 310.37 last 51.16) - open/close/unlink 13847 (time 1788361245.94 total 320.38 last 44.88) - open/close/unlink 14247 (time 1788361255.94 total 330.38 last 39.99) - open/close/unlink 14646 (time 1788361266.00 total 340.43 last 39.68) - open/close/unlink 15075 (time 1788361276.01 total 350.45 last 42.84) - open/close/unlink 15508 (time 1788361286.04 total 360.47 last 43.19) - open/close/unlink 15927 (time 1788361296.05 total 370.48 last 41.86) - open/close/unlink 16420 (time 1788361306.06 total 380.49 last 49.25) - open/close/unlink 16922 (time 1788361316.07 total 390.51 last 50.12) - open/close/unlink 17419 (time 1788361326.08 total 400.51 last 49.68) - open/close/unlink 17880 (time 1788361336.10 total 410.53 last 46.02) - open/close/unlink 18336 (time 1788361346.10 total 420.53 last 45.58) - open/close/unlink 18772 (time 1788361356.11 total 430.55 last 43.54) - open/close/unlink 19235 (time 1788361366.12 total 440.56 last 46.26) - open/close/unlink 19702 (time 1788361376.13 total 450.57 last 46.65) - open/close/unlink 20000 (time 1788361382.65 total 457.08 last 45.76) - open/close/unlink 20497 (time 1788361392.65 total 467.08 last 49.69) - open/close/unlink 21038 (time 1788361402.65 total 477.09 last 54.08) - open/close/unlink 21554 (time 1788361412.67 total 487.11 last 51.49) - open/close/unlink 22001 (time 1788361422.68 total 497.11 last 44.70) - open/close/unlink 22527 (time 1788361432.69 total 507.12 last 52.54) - open/close/unlink 23120 (time 1788361442.71 total 517.14 last 59.18) - open/close/unlink 23580 (time 1788361452.71 total 527.14 last 45.99) - open/close/unlink 24098 (time 1788361462.72 total 537.15 last 51.74) - open/close/unlink 24708 (time 1788361472.73 total 547.17 last 60.92) - open/close/unlink 25410 (time 1788361482.74 total 557.17 last 70.17) - open/close/unlink 26058 (time 1788361492.74 total 567.17 last 64.80) - open/close/unlink 26577 (time 1788361501.91 total 575.83 last 51.96) - open/close/unlink 27144 (time 1788361511.92 total 585.84 last 56.84) - open/close/unlink 27725 (time 1788361521.95 total 595.87 last 57.92) - open/close/unlink 28239 (time 1788361531.97 total 605.88 last 51.31) - open/close/unlink 28626 (time 1788361541.97 total 615.89 last 38.70) - open/close/unlink 29234 (time 1788361551.99 total 625.90 last 60.69) - open/close/unlink 29777 (time 1788361562.00 total 635.92 last 54.23) - open/close/unlink 30000 (time 1788361566.08 total 639.99 last 54.69) - open/close/unlink 30560 (time 1788361576.08 total 650.00 last 55.98) - open/close/unlink 31203 (time 1788361586.09 total 660.00 last 64.27) - open/close/unlink 31854 (time 1788361596.13 total 670.04 last 64.85) - open/close/unlink 32427 (time 1788361606.13 total 680.04 last 57.29) - open/close/unlink 32993 (time 1788361616.15 total 690.06 last 56.50) - open/close/unlink 33520 (time 1788361626.16 total 700.08 last 52.63) - open/close/unlink 34133 (time 1788361636.17 total 710.09 last 61.22) - open/close/unlink 34677 (time 1788361646.18 total 720.09 last 54.38) - open/close/unlink 35294 (time 1788361656.20 total 730.11 last 61.57) - open/close/unlink 35816 (time 1788361666.22 total 740.13 last 52.12) - open/close/unlink 36467 (time 1788361676.22 total 750.13 last 65.10) - open/close/unlink 37246 (time 1788361686.22 total 760.13 last 77.89) - open/close/unlink 37929 (time 1788361696.22 total 770.13 last 68.30) - open/close/unlink 38513 (time 1788361706.23 total 780.14 last 58.35) - open/close/unlink 39198 (time 1788361716.23 total 790.15 last 68.45) - open/close/unlink 39810 (time 1788361726.23 total 800.15 last 61.19) - open/close/unlink 40000 (time 1788361729.62 total 803.54 last 56.09) - open/close/unlink 40621 (time 1788361739.62 total 813.54 last 62.08) - open/close/unlink 41186 (time 1788361749.65 total 823.56 last 56.38) - open/close/unlink 41699 (time 1788361759.67 total 833.58 last 51.19) - open/close/unlink 42186 (time 1788361769.69 total 843.61 last 48.59) - open/close/unlink 42645 (time 1788361779.70 total 853.62 last 45.84) - open/close/unlink 43088 (time 1788361789.72 total 863.63 last 44.24) - open/close/unlink 43587 (time 1788361799.72 total 873.64 last 49.86) - open/close/unlink 44200 (time 1788361809.73 total 883.64 last 61.28) - open/close/unlink 44956 (time 1788361819.73 total 893.65 last 75.55) - open/close/unlink 45524 (time 1788361829.74 total 903.65 last 56.77) - open/close/unlink 46163 (time 1788361839.75 total 913.66 last 63.84) - open/close/unlink 46823 (time 1788361849.75 total 923.66 last 65.99) - open/close/unlink 47706 (time 1788361859.76 total 933.68 last 88.17) - open/close/unlink 48494 (time 1788361869.77 total 943.68 last 78.78) - open/close/unlink 49359 (time 1788361879.77 total 953.68 last 86.49) - open/close/unlink 50000 (time 1788361886.27 total 960.18 last 98.62) - open/close/unlink 51260 (time 1788361896.27 total 970.19 last 125.92) - open/close/unlink 52386 (time 1788361906.28 total 980.19 last 112.55) - open/close/unlink 53702 (time 1788361916.28 total 990.19 last 131.60) - open/close/unlink 54769 (time 1788361926.28 total 1000.19 last 106.68) - open/close/unlink 55776 (time 1788361936.28 total 1010.20 last 100.69) - open/close/unlink 56724 (time 1788361946.28 total 1020.20 last 94.77) - open/close/unlink 57680 (time 1788361956.29 total 1030.21 last 95.52) - open/close/unlink 58811 (time 1788361966.29 total 1040.21 last 113.10) - open/close/unlink 59840 (time 1788361976.30 total 1050.21 last 102.88) - open/close/unlink 61070 (time 1788361986.30 total 1060.21 last 122.99) - open/close/unlink 62212 (time 1788361996.30 total 1070.21 last 114.19) - open/close/unlink 63118 (time 1788362006.30 total 1080.22 last 90.57) - open/close/unlink 64073 (time 1788362016.31 total 1090.22 last 95.44) total: 64768 open/close/unlink in 1096.75 seconds: 59.05 ops/second (time 1788361502.04 total 575.94 last 52.11) - open/close/unlink 27180 (time 1788361512.05 total 585.94 last 57.36) - open/close/unlink 27763 (time 1788361522.05 total 595.94 last 58.30) - open/close/unlink 28276 (time 1788361532.06 total 605.95 last 51.25) - open/close/unlink 28662 (time 1788361542.07 total 615.96 last 38.58) - open/close/unlink 29258 (time 1788361552.07 total 625.96 last 59.57) - open/close/unlink 29790 (time 1788361562.08 total 635.98 last 53.14) - open/close/unlink 30000 (time 1788361565.99 total 639.88 last 53.78) - open/close/unlink 30564 (time 1788361575.99 total 649.88 last 56.39) - open/close/unlink 31209 (time 1788361586.00 total 659.89 last 64.48) - open/close/unlink 31871 (time 1788361596.00 total 669.90 last 66.14) - open/close/unlink 32439 (time 1788361606.02 total 679.91 last 56.70) - open/close/unlink 33002 (time 1788361616.03 total 689.92 last 56.28) - open/close/unlink 33530 (time 1788361626.04 total 699.93 last 52.75) - open/close/unlink 34137 (time 1788361636.05 total 709.94 last 60.63) - open/close/unlink 34681 (time 1788361646.05 total 719.95 last 54.36) - open/close/unlink 35299 (time 1788361656.06 total 729.96 last 61.74) - open/close/unlink 35826 (time 1788361666.07 total 739.96 last 52.66) - open/close/unlink 36459 (time 1788361676.08 total 749.97 last 63.26) - open/close/unlink 37241 (time 1788361686.08 total 759.97 last 78.18) - open/close/unlink 37920 (time 1788361696.09 total 769.98 last 67.84) - open/close/unlink 38507 (time 1788361706.09 total 779.98 last 58.68) - open/close/unlink 39186 (time 1788361716.10 total 790.00 last 67.81) - open/close/unlink 39795 (time 1788361726.12 total 800.01 last 60.82) - open/close/unlink 40000 (time 1788361729.78 total 803.68 last 55.93) - open/close/unlink 40620 (time 1788361739.79 total 813.69 last 61.94) - open/close/unlink 41178 (time 1788361749.81 total 823.70 last 55.72) - open/close/unlink 41690 (time 1788361759.82 total 833.71 last 51.14) - open/close/unlink 42159 (time 1788361769.86 total 843.75 last 46.73) - open/close/unlink 42620 (time 1788361779.86 total 853.75 last 46.07) - open/close/unlink 43063 (time 1788361789.87 total 863.76 last 44.26) - open/close/unlink 43562 (time 1788361799.88 total 873.77 last 49.87) - open/close/unlink 44182 (time 1788361809.88 total 883.78 last 61.95) - open/close/unlink 44947 (time 1788361819.89 total 893.79 last 76.42) - open/close/unlink 45511 (time 1788361829.91 total 903.80 last 56.31) - open/close/unlink 46156 (time 1788361839.92 total 913.81 last 64.45) - open/close/unlink 46817 (time 1788361849.92 total 923.81 last 66.06) - open/close/unlink 47700 (time 1788361859.93 total 933.82 last 88.26) - open/close/unlink 48487 (time 1788361869.94 total 943.83 last 78.59) - open/close/unlink 49352 (time 1788361879.95 total 953.84 last 86.42) - open/close/unlink 50000 (time 1788361886.41 total 960.31 last 100.26) - open/close/unlink 51258 (time 1788361896.42 total 970.31 last 125.78) - open/close/unlink 52403 (time 1788361906.42 total 980.31 last 114.50) - open/close/unlink 53721 (time 1788361916.42 total 990.31 last 131.78) - open/close/unlink 54786 (time 1788361926.42 total 1000.32 last 106.42) - open/close/unlink 55788 (time 1788361936.43 total 1010.32 last 100.15) - open/close/unlink 56732 (time 1788361946.43 total 1020.32 last 94.40) - open/close/unlink 57689 (time 1788361956.43 total 1030.33 last 95.67) - open/close/unlink 58826 (time 1788361966.44 total 1040.33 last 113.64) - open/close/unlink 59847 (time 1788361976.44 total 1050.33 last 102.06) - open/close/unlink 61080 (time 1788361986.44 total 1060.34 last 123.29) - open/close/unlink 62220 (time 1788361996.45 total 1070.34 last 113.93) - open/close/unlink 63132 (time 1788362006.45 total 1080.34 last 91.18) - open/close/unlink 64073 (time 1788362016.46 total 1090.35 last 94.07) total: 64768 open/close/unlink in 1097.07 seconds: 59.04 ops/second (time 1788361502.14 total 576.03 last 51.62) - open/close/unlink 27150 (time 1788361512.14 total 586.03 last 57.75) - open/close/unlink 27738 (time 1788361522.16 total 596.05 last 58.70) - open/close/unlink 28255 (time 1788361532.17 total 606.06 last 51.63) - open/close/unlink 28644 (time 1788361542.19 total 616.08 last 38.85) - open/close/unlink 29263 (time 1788361552.19 total 626.08 last 61.90) - open/close/unlink 29798 (time 1788361562.19 total 636.09 last 53.46) - open/close/unlink 30000 (time 1788361565.92 total 639.81 last 54.18) - open/close/unlink 30555 (time 1788361575.93 total 649.82 last 55.48) - open/close/unlink 31205 (time 1788361585.93 total 659.82 last 64.96) - open/close/unlink 31871 (time 1788361595.95 total 669.84 last 66.48) - open/close/unlink 32433 (time 1788361605.96 total 679.85 last 56.18) - open/close/unlink 33004 (time 1788361615.96 total 689.85 last 57.10) - open/close/unlink 33538 (time 1788361625.96 total 699.85 last 53.38) - open/close/unlink 34148 (time 1788361635.98 total 709.87 last 60.90) - open/close/unlink 34693 (time 1788361645.99 total 719.88 last 54.45) - open/close/unlink 35304 (time 1788361656.00 total 729.89 last 61.03) - open/close/unlink 35827 (time 1788361666.01 total 739.90 last 52.24) - open/close/unlink 36463 (time 1788361676.03 total 749.92 last 63.48) - open/close/unlink 37250 (time 1788361686.03 total 759.92 last 78.69) - open/close/unlink 37931 (time 1788361696.04 total 769.93 last 68.04) - open/close/unlink 38514 (time 1788361706.04 total 779.93 last 58.28) - open/close/unlink 39192 (time 1788361716.04 total 789.93 last 67.77) - open/close/unlink 39808 (time 1788361726.07 total 799.96 last 61.46) - open/close/unlink 40000 (time 1788361729.45 total 803.34 last 56.74) - open/close/unlink 40608 (time 1788361739.46 total 813.35 last 60.73) - open/close/unlink 41175 (time 1788361749.46 total 823.35 last 56.69) - open/close/unlink 41689 (time 1788361759.48 total 833.37 last 51.33) - open/close/unlink 42161 (time 1788361769.50 total 843.39 last 47.10) - open/close/unlink 42619 (time 1788361779.51 total 853.40 last 45.73) - open/close/unlink 43059 (time 1788361789.52 total 863.41 last 43.99) - open/close/unlink 43552 (time 1788361799.52 total 873.41 last 49.28) - open/close/unlink 44154 (time 1788361809.52 total 883.41 last 60.19) - open/close/unlink 44913 (time 1788361819.54 total 893.43 last 75.77) - open/close/unlink 45472 (time 1788361829.54 total 903.43 last 55.90) - open/close/unlink 46117 (time 1788361839.55 total 913.44 last 64.44) - open/close/unlink 46775 (time 1788361849.55 total 923.44 last 65.79) - open/close/unlink 47645 (time 1788361859.56 total 933.45 last 86.90) - open/close/unlink 48422 (time 1788361869.56 total 943.45 last 77.69) - open/close/unlink 49301 (time 1788361879.58 total 953.47 last 87.72) - open/close/unlink 50000 (time 1788361886.67 total 960.56 last 98.71) - open/close/unlink 51245 (time 1788361896.67 total 970.57 last 124.39) - open/close/unlink 52406 (time 1788361906.67 total 980.57 last 116.10) - open/close/unlink 53707 (time 1788361916.68 total 990.57 last 129.99) - open/close/unlink 54762 (time 1788361926.69 total 1000.58 last 105.41) - open/close/unlink 55769 (time 1788361936.70 total 1010.59 last 100.65) - open/close/unlink 56734 (time 1788361946.71 total 1020.60 last 96.37) - open/close/unlink 57685 (time 1788361956.72 total 1030.61 last 95.04) - open/close/unlink 58843 (time 1788361966.72 total 1040.61 last 115.74) - open/close/unlink 59874 (time 1788361976.73 total 1050.62 last 103.04) - open/close/unlink 61105 (time 1788361986.73 total 1060.62 last 123.02) - open/close/unlink 62241 (time 1788361996.74 total 1070.63 last 113.55) - open/close/unlink 63151 (time 1788362006.75 total 1080.64 last 90.86) - open/close/unlink 64108 (time 1788362016.76 total 1090.65 last 95.64) total: 64768 open/close/unlink in 1097.19 seconds: 59.03 ops/second (time 1788361501.81 total 575.98 last 52.57) - open/close/unlink 27126 (time 1788361511.82 total 585.98 last 55.75) - open/close/unlink 27719 (time 1788361521.83 total 596.00 last 59.23) - open/close/unlink 28226 (time 1788361531.83 total 606.00 last 50.69) - open/close/unlink 28623 (time 1788361541.83 total 616.00 last 39.70) - open/close/unlink 29226 (time 1788361551.84 total 626.01 last 60.23) - open/close/unlink 29769 (time 1788361561.85 total 636.02 last 54.26) - open/close/unlink 30000 (time 1788361566.05 total 640.22 last 54.97) - open/close/unlink 30557 (time 1788361576.06 total 650.23 last 55.67) - open/close/unlink 31197 (time 1788361586.06 total 660.23 last 63.97) - open/close/unlink 31852 (time 1788361596.08 total 670.25 last 65.36) - open/close/unlink 32414 (time 1788361606.10 total 680.26 last 56.13) - open/close/unlink 32980 (time 1788361616.10 total 690.26 last 56.59) - open/close/unlink 33504 (time 1788361626.10 total 700.27 last 52.38) - open/close/unlink 34117 (time 1788361636.11 total 710.27 last 61.27) - open/close/unlink 34661 (time 1788361646.11 total 720.28 last 54.38) - open/close/unlink 35274 (time 1788361656.12 total 730.29 last 61.23) - open/close/unlink 35795 (time 1788361666.13 total 740.29 last 52.08) - open/close/unlink 36435 (time 1788361676.13 total 750.30 last 63.98) - open/close/unlink 37221 (time 1788361686.14 total 760.30 last 78.54) - open/close/unlink 37901 (time 1788361696.14 total 770.31 last 67.99) - open/close/unlink 38482 (time 1788361706.15 total 780.31 last 58.07) - open/close/unlink 39161 (time 1788361716.15 total 790.32 last 67.86) - open/close/unlink 39771 (time 1788361726.16 total 800.32 last 60.97) - open/close/unlink 40000 (time 1788361730.11 total 804.27 last 57.95) - open/close/unlink 40621 (time 1788361740.11 total 814.28 last 62.05) - open/close/unlink 41181 (time 1788361750.12 total 824.28 last 55.99) - open/close/unlink 41692 (time 1788361760.14 total 834.31 last 50.98) - open/close/unlink 42164 (time 1788361770.15 total 844.32 last 47.14) - open/close/unlink 42622 (time 1788361780.17 total 854.34 last 45.71) - open/close/unlink 43076 (time 1788361790.20 total 864.37 last 45.28) - open/close/unlink 43587 (time 1788361800.22 total 874.39 last 51.01) - open/close/unlink 44205 (time 1788361810.23 total 884.40 last 61.72) - open/close/unlink 44960 (time 1788361820.24 total 894.41 last 75.41) - open/close/unlink 45519 (time 1788361830.25 total 904.42 last 55.85) - open/close/unlink 46167 (time 1788361840.27 total 914.44 last 64.68) - open/close/unlink 46837 (time 1788361850.28 total 924.45 last 66.96) - open/close/unlink 47705 (time 1788361860.29 total 934.45 last 86.74) - open/close/unlink 48508 (time 1788361870.29 total 944.46 last 80.26) - open/close/unlink 49353 (time 1788361880.31 total 954.47 last 84.37) - open/close/unlink 50000 (time 1788361886.67 total 960.84 last 101.63) - open/close/unlink 51244 (time 1788361896.68 total 970.84 last 124.34) - open/close/unlink 52410 (time 1788361906.68 total 980.85 last 116.52) - open/close/unlink 53712 (time 1788361916.69 total 990.86 last 130.14) - open/close/unlink 54763 (time 1788361926.70 total 1000.87 last 105.00) - open/close/unlink 55768 (time 1788361936.70 total 1010.87 last 100.49) - open/close/unlink 56719 (time 1788361946.70 total 1020.87 last 95.09) - open/close/unlink 57675 (time 1788361956.70 total 1030.87 last 95.60) - open/close/unlink 58814 (time 1788361966.71 total 1040.87 last 113.82) - open/close/unlink 59850 (time 1788361976.71 total 1050.87 last 103.60) - open/close/unlink 61087 (time 1788361986.71 total 1060.88 last 123.64) - open/close/unlink 62225 (time 1788361996.71 total 1070.88 last 113.78) - open/close/unlink 63134 (time 1788362006.72 total 1080.89 last 90.84) - open/close/unlink 64090 (time 1788362016.72 total 1090.89 last 95.58) total: 64768 open/close/unlink in 1097.71 seconds: 59.00 ops/second (time 1788361502.74 total 577.17 last 51.88) - open/close/unlink 27171 (time 1788361512.75 total 587.18 last 59.35) - open/close/unlink 27747 (time 1788361522.78 total 597.21 last 57.42) - open/close/unlink 28257 (time 1788361532.78 total 607.21 last 50.99) - open/close/unlink 28657 (time 1788361542.80 total 617.23 last 39.92) - open/close/unlink 29267 (time 1788361552.82 total 627.26 last 60.87) - open/close/unlink 29794 (time 1788361562.83 total 637.26 last 52.68) - open/close/unlink 30000 (time 1788361566.76 total 641.20 last 52.33) - open/close/unlink 30562 (time 1788361576.79 total 651.23 last 56.03) - open/close/unlink 31230 (time 1788361586.80 total 661.23 last 66.76) - open/close/unlink 31865 (time 1788361596.81 total 671.24 last 63.44) - open/close/unlink 32434 (time 1788361606.81 total 681.24 last 56.89) - open/close/unlink 32999 (time 1788361616.81 total 691.25 last 56.48) - open/close/unlink 33514 (time 1788361626.82 total 701.25 last 51.48) - open/close/unlink 34131 (time 1788361636.83 total 711.26 last 61.65) - open/close/unlink 34683 (time 1788361646.86 total 721.29 last 55.03) - open/close/unlink 35301 (time 1788361656.88 total 731.31 last 61.68) - open/close/unlink 35834 (time 1788361666.88 total 741.32 last 53.25) - open/close/unlink 36510 (time 1788361676.89 total 751.32 last 67.58) - open/close/unlink 37265 (time 1788361686.90 total 761.33 last 75.42) - open/close/unlink 37946 (time 1788361696.91 total 771.34 last 68.03) - open/close/unlink 38544 (time 1788361706.91 total 781.34 last 59.79) - open/close/unlink 39219 (time 1788361716.91 total 791.35 last 67.47) - open/close/unlink 39812 (time 1788361726.94 total 801.38 last 59.14) - open/close/unlink 40000 (time 1788361730.08 total 804.52 last 59.87) - open/close/unlink 40620 (time 1788361740.10 total 814.53 last 61.91) - open/close/unlink 41183 (time 1788361750.11 total 824.54 last 56.25) - open/close/unlink 41699 (time 1788361760.12 total 834.55 last 51.52) - open/close/unlink 42173 (time 1788361770.13 total 844.57 last 47.34) - open/close/unlink 42639 (time 1788361780.14 total 854.57 last 46.56) - open/close/unlink 43081 (time 1788361790.14 total 864.58 last 44.20) - open/close/unlink 43584 (time 1788361800.15 total 874.58 last 50.26) - open/close/unlink 44200 (time 1788361810.15 total 884.59 last 61.58) - open/close/unlink 44955 (time 1788361820.16 total 894.60 last 75.41) - open/close/unlink 45523 (time 1788361830.18 total 904.61 last 56.71) - open/close/unlink 46166 (time 1788361840.18 total 914.61 last 64.29) - open/close/unlink 46834 (time 1788361850.19 total 924.62 last 66.77) - open/close/unlink 47708 (time 1788361860.19 total 934.62 last 87.37) - open/close/unlink 48492 (time 1788361870.19 total 944.63 last 78.35) - open/close/unlink 49338 (time 1788361880.21 total 954.64 last 84.47) - open/close/unlink 50000 (time 1788361886.69 total 961.12 last 102.22) - open/close/unlink 51242 (time 1788361896.69 total 971.12 last 124.16) - open/close/unlink 52409 (time 1788361906.69 total 981.13 last 116.66) - open/close/unlink 53694 (time 1788361916.70 total 991.13 last 128.43) - open/close/unlink 54742 (time 1788361926.70 total 1001.14 last 104.74) - open/close/unlink 55750 (time 1788361936.71 total 1011.14 last 100.75) - open/close/unlink 56705 (time 1788361946.71 total 1021.14 last 95.48) - open/close/unlink 57653 (time 1788361956.71 total 1031.15 last 94.76) - open/close/unlink 58809 (time 1788361966.72 total 1041.16 last 115.51) - open/close/unlink 59838 (time 1788361976.72 total 1051.16 last 102.87) - open/close/unlink 61068 (time 1788361986.73 total 1061.17 last 122.91) - open/close/unlink 62202 (time 1788361996.73 total 1071.17 last 113.40) - open/close/unlink 63114 (time 1788362006.74 total 1081.17 last 91.13) - open/close/unlink 64062 (time 1788362016.74 total 1091.18 last 94.77) total: 64768 open/close/unlink in 1098.69 seconds: 58.95 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg228-server: cannot open 'lustre-ost2': no such pool pdsh@oleg228-client: oleg228-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg228-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg228-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg228-server stop mds service on oleg228-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg228-server unloading modules via unload_modules_local on: 'oleg228-server' oleg228-server: oleg228-server.virtnet: executing unload_modules_local oleg228-server: modules unloaded. pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-client: ssh exited with exit code 2 pdsh@oleg228-client: oleg228-server: ssh exited with exit code 2