== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:25:01 (1788791101) start mds service on oleg639-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg639-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg639-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg639-server: oleg639-server.virtnet: executing set_default_debug -1 all pdsh@oleg639-client: oleg639-server: ssh exited with exit code 1 Started lustre-OST0000 oleg639-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg639-client.virtnet: -o user_xattr,flock 192.168.206.139@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 710 (time 1788791136.52 total 10.01 last 70.95) - open/close/unlink 1335 (time 1788791146.52 total 20.01 last 62.50) - open/close/unlink 2181 (time 1788791156.53 total 30.02 last 84.53) - open/close/unlink 3111 (time 1788791166.53 total 40.02 last 93.00) - open/close/unlink 4064 (time 1788791176.53 total 50.02 last 95.28) - open/close/unlink 5006 (time 1788791186.54 total 60.03 last 94.13) - open/close/unlink 5882 (time 1788791196.55 total 70.03 last 87.55) - open/close/unlink 6757 (time 1788791206.55 total 80.04 last 87.46) - open/close/unlink 7620 (time 1788791216.55 total 90.04 last 86.29) - open/close/unlink 8460 (time 1788791226.56 total 100.05 last 83.90) - open/close/unlink 9544 (time 1788791236.58 total 110.06 last 108.22) - open/close/unlink 10000 (time 1788791241.41 total 114.89 last 94.43) - open/close/unlink 10939 (time 1788791251.42 total 124.90 last 93.84) - open/close/unlink 11912 (time 1788791261.42 total 134.90 last 97.28) - open/close/unlink 12801 (time 1788791271.42 total 144.91 last 88.86) - open/close/unlink 13716 (time 1788791281.42 total 154.91 last 91.47) - open/close/unlink 14570 (time 1788791291.43 total 164.91 last 85.38) - open/close/unlink 15285 (time 1788791301.44 total 174.92 last 71.41) - open/close/unlink 16080 (time 1788791311.46 total 184.94 last 79.38) - open/close/unlink 16759 (time 1788791321.46 total 194.94 last 67.88) - open/close/unlink 17516 (time 1788791331.47 total 204.95 last 75.64) - open/close/unlink 18292 (time 1788791341.47 total 214.96 last 77.54) - open/close/unlink 19042 (time 1788791351.48 total 224.96 last 74.96) - open/close/unlink 19758 (time 1788791361.48 total 234.96 last 71.60) - open/close/unlink 20000 (time 1788791364.19 total 237.67 last 89.31) - open/close/unlink 20761 (time 1788791374.19 total 247.67 last 76.08) - open/close/unlink 21520 (time 1788791384.20 total 257.68 last 75.83) - open/close/unlink 22249 (time 1788791394.21 total 267.69 last 72.82) - open/close/unlink 23017 (time 1788791404.21 total 277.70 last 76.79) - open/close/unlink 23821 (time 1788791414.22 total 287.70 last 80.34) - open/close/unlink 24837 (time 1788791424.22 total 297.70 last 101.59) - open/close/unlink 25796 (time 1788791434.23 total 307.71 last 95.84) - open/close/unlink 26693 (time 1788791444.23 total 317.72 last 89.64) - open/close/unlink 27330 (time 1788791454.24 total 327.72 last 63.66) - open/close/unlink 28202 (time 1788791464.25 total 337.73 last 87.12) - open/close/unlink 29093 (time 1788791474.25 total 347.73 last 89.08) - open/close/unlink 30000 (time 1788791483.78 total 357.27 last 95.13) - open/close/unlink 30812 (time 1788791493.80 total 367.28 last 81.11) - open/close/unlink 31683 (time 1788791503.80 total 377.28 last 87.09) - open/close/unlink 32523 (time 1788791513.80 total 387.28 last 83.99) - open/close/unlink 33418 (time 1788791523.81 total 397.29 last 89.42) - open/close/unlink 34299 (time 1788791533.82 total 407.30 last 88.01) - open/close/unlink 35312 (time 1788791543.82 total 417.31 last 101.27) - open/close/unlink 36177 (time 1788791553.83 total 427.31 last 86.43) - open/close/unlink 37208 (time 1788791563.83 total 437.32 last 103.06) - open/close/unlink 38126 (time 1788791573.83 total 447.32 last 91.78) - open/close/unlink 39062 (time 1788791583.84 total 457.32 last 93.57) - open/close/unlink 39833 (time 1788791593.84 total 467.32 last 77.08) - open/close/unlink 40000 (time 1788791595.90 total 469.39 last 80.95) - open/close/unlink 40822 (time 1788791605.91 total 479.39 last 82.17) - open/close/unlink 41591 (time 1788791615.91 total 489.40 last 76.84) - open/close/unlink 42481 (time 1788791625.92 total 499.40 last 88.97) - open/close/unlink 43496 (time 1788791635.92 total 509.41 last 101.43) - open/close/unlink 44533 (time 1788791645.93 total 519.41 last 103.64) - open/close/unlink 45564 (time 1788791655.94 total 529.42 last 103.02) - open/close/unlink 46611 (time 1788791665.94 total 539.42 last 104.68) - open/close/unlink 47231 (time 1788791675.95 total 549.44 last 61.93) - op - open/close/unlink 707 (time 1788791136.54 total 10.01 last 70.63) - open/close/unlink 1335 (time 1788791146.56 total 20.03 last 62.69) - open/close/unlink 2194 (time 1788791156.57 total 30.03 last 85.86) - open/close/unlink 3126 (time 1788791166.58 total 40.04 last 93.11) - open/close/unlink 4082 (time 1788791176.58 total 50.04 last 95.60) - open/close/unlink 5026 (time 1788791186.58 total 60.05 last 94.36) - open/close/unlink 5897 (time 1788791196.59 total 70.05 last 87.03) - open/close/unlink 6778 (time 1788791206.59 total 80.06 last 88.07) - open/close/unlink 7645 (time 1788791216.60 total 90.06 last 86.68) - open/close/unlink 8476 (time 1788791226.60 total 100.07 last 83.04) - open/close/unlink 9553 (time 1788791236.61 total 110.08 last 107.61) - open/close/unlink 10000 (time 1788791241.31 total 114.77 last 95.21) - open/close/unlink 10937 (time 1788791251.31 total 124.77 last 93.67) - open/close/unlink 11905 (time 1788791261.32 total 134.78 last 96.71) - open/close/unlink 12789 (time 1788791271.32 total 144.79 last 88.35) - open/close/unlink 13705 (time 1788791281.33 total 154.79 last 91.58) - open/close/unlink 14565 (time 1788791291.34 total 164.80 last 85.92) - open/close/unlink 15284 (time 1788791301.34 total 174.81 last 71.87) - open/close/unlink 16076 (time 1788791311.35 total 184.82 last 79.13) - open/close/unlink 16751 (time 1788791321.36 total 194.82 last 67.45) - open/close/unlink 17498 (time 1788791331.36 total 204.83 last 74.64) - open/close/unlink 18262 (time 1788791341.37 total 214.83 last 76.37) - open/close/unlink 19020 (time 1788791351.38 total 224.85 last 75.71) - open/close/unlink 19745 (time 1788791361.39 total 234.85 last 72.45) - open/close/unlink 20000 (time 1788791364.27 total 237.74 last 88.47) - open/close/unlink 20759 (time 1788791374.27 total 247.74 last 75.88) - open/close/unlink 21515 (time 1788791384.29 total 257.75 last 75.50) - open/close/unlink 22248 (time 1788791394.30 total 267.76 last 73.23) - open/close/unlink 23023 (time 1788791404.30 total 277.76 last 77.49) - open/close/unlink 23831 (time 1788791414.30 total 287.76 last 80.79) - open/close/unlink 24850 (time 1788791424.30 total 297.77 last 101.86) - open/close/unlink 25806 (time 1788791434.31 total 307.77 last 95.53) - open/close/unlink 26706 (time 1788791444.31 total 317.78 last 89.97) - open/close/unlink 27339 (time 1788791454.31 total 327.78 last 63.28) - open/close/unlink 28204 (time 1788791464.32 total 337.79 last 86.41) - open/close/unlink 29105 (time 1788791474.34 total 347.80 last 89.98) - open/close/unlink 30000 (time 1788791483.83 total 357.30 last 94.26) - open/close/unlink 30806 (time 1788791493.84 total 367.30 last 80.57) - open/close/unlink 31667 (time 1788791503.84 total 377.30 last 86.09) - open/close/unlink 32502 (time 1788791513.84 total 387.30 last 83.49) - open/close/unlink 33404 (time 1788791523.85 total 397.32 last 90.08) - open/close/unlink 34282 (time 1788791533.86 total 407.32 last 87.74) - open/close/unlink 35303 (time 1788791543.86 total 417.33 last 102.07) - open/close/unlink 36165 (time 1788791553.87 total 427.33 last 86.15) - open/close/unlink 37190 (time 1788791563.87 total 437.34 last 102.44) - open/close/unlink 38116 (time 1788791573.88 total 447.34 last 92.56) - open/close/unlink 39050 (time 1788791583.89 total 457.35 last 93.30) - open/close/unlink 39830 (time 1788791593.90 total 467.36 last 77.92) - open/close/unlink 40000 (time 1788791596.01 total 469.47 last 80.66) - open/close/unlink 40822 (time 1788791606.01 total 479.48 last 82.14) - open/close/unlink 41591 (time 1788791616.02 total 489.49 last 76.85) - open/close/unlink 42489 (time 1788791626.04 total 499.50 last 89.66) - open/close/unlink 43507 (time 1788791636.04 total 509.50 last 101.78) - open/close/unlink 44536 (time 1788791646.04 total 519.50 last 102.88) - open/close/unlink 45567 (time 1788791656.04 total 529.51 last 103.05) - open/close/unlink 46621 (time 1788791666.05 total 539.52 last 105.30) - open/close/unlink 47225 (time 1788791676.06 total 549.52 last 60.39) - op - open/close/unlink 709 (time 1788791136.59 total 10.01 last 70.85) - open/close/unlink 1333 (time 1788791146.62 total 20.03 last 62.26) - open/close/unlink 2187 (time 1788791156.62 total 30.03 last 85.36) - open/close/unlink 3124 (time 1788791166.62 total 40.04 last 93.66) - open/close/unlink 4076 (time 1788791176.63 total 50.04 last 95.18) - open/close/unlink 5025 (time 1788791186.63 total 60.04 last 94.88) - open/close/unlink 5893 (time 1788791196.64 total 70.05 last 86.74) - open/close/unlink 6771 (time 1788791206.64 total 80.05 last 87.78) - open/close/unlink 7635 (time 1788791216.64 total 90.06 last 86.34) - open/close/unlink 8463 (time 1788791226.66 total 100.07 last 82.70) - open/close/unlink 9540 (time 1788791236.66 total 110.08 last 107.65) - open/close/unlink 10000 (time 1788791241.45 total 114.86 last 96.05) - open/close/unlink 10933 (time 1788791251.46 total 124.87 last 93.24) - open/close/unlink 11903 (time 1788791261.46 total 134.87 last 96.98) - open/close/unlink 12782 (time 1788791271.47 total 144.88 last 87.86) - open/close/unlink 13702 (time 1788791281.47 total 154.88 last 91.98) - open/close/unlink 14552 (time 1788791291.47 total 164.88 last 85.00) - open/close/unlink 15274 (time 1788791301.47 total 174.89 last 72.16) - open/close/unlink 16075 (time 1788791311.48 total 184.89 last 80.04) - open/close/unlink 16754 (time 1788791321.49 total 194.90 last 67.84) - open/close/unlink 17494 (time 1788791331.50 total 204.91 last 73.91) - open/close/unlink 18268 (time 1788791341.51 total 214.93 last 77.30) - open/close/unlink 19023 (time 1788791351.52 total 224.93 last 75.45) - open/close/unlink 19749 (time 1788791361.53 total 234.94 last 72.56) - open/close/unlink 20000 (time 1788791364.38 total 237.79 last 87.96) - open/close/unlink 20759 (time 1788791374.39 total 247.80 last 75.82) - open/close/unlink 21519 (time 1788791384.40 total 257.82 last 75.90) - open/close/unlink 22259 (time 1788791394.41 total 267.82 last 73.96) - open/close/unlink 23035 (time 1788791404.42 total 277.83 last 77.51) - open/close/unlink 23838 (time 1788791414.43 total 287.84 last 80.25) - open/close/unlink 24850 (time 1788791424.43 total 297.85 last 101.13) - open/close/unlink 25812 (time 1788791434.44 total 307.85 last 96.13) - open/close/unlink 26703 (time 1788791444.45 total 317.86 last 89.02) - open/close/unlink 27345 (time 1788791454.45 total 327.87 last 64.18) - open/close/unlink 28217 (time 1788791464.46 total 337.87 last 87.15) - open/close/unlink 29110 (time 1788791474.47 total 347.88 last 89.18) - open/close/unlink 30000 (time 1788791483.85 total 357.26 last 94.91) - open/close/unlink 30815 (time 1788791493.86 total 367.27 last 81.42) - open/close/unlink 31683 (time 1788791503.86 total 377.27 last 86.79) - open/close/unlink 32521 (time 1788791513.87 total 387.28 last 83.74) - open/close/unlink 33423 (time 1788791523.87 total 397.28 last 90.19) - open/close/unlink 34301 (time 1788791533.87 total 407.28 last 87.77) - open/close/unlink 35311 (time 1788791543.88 total 417.29 last 100.96) - open/close/unlink 36179 (time 1788791553.88 total 427.29 last 86.77) - open/close/unlink 37215 (time 1788791563.88 total 437.29 last 103.57) - open/close/unlink 38126 (time 1788791573.89 total 447.30 last 91.05) - open/close/unlink 39056 (time 1788791583.89 total 457.31 last 92.95) - open/close/unlink 39827 (time 1788791593.90 total 467.32 last 77.01) - open/close/unlink 40000 (time 1788791596.02 total 469.43 last 81.94) - open/close/unlink 40827 (time 1788791606.02 total 479.43 last 82.65) - open/close/unlink 41594 (time 1788791616.03 total 489.44 last 76.65) - open/close/unlink 42491 (time 1788791626.03 total 499.45 last 89.65) - open/close/unlink 43513 (time 1788791636.05 total 509.46 last 102.09) - open/close/unlink 44537 (time 1788791646.05 total 519.46 last 102.37) - open/close/unlink 45569 (time 1788791656.05 total 529.46 last 103.19) - open/close/unlink 46628 (time 1788791666.05 total 539.47 last 105.85) - open/close/unlink 47235 (time 1788791676.07 total 549.48 last 60.62) - op - open/close/unlink 708 (time 1788791136.69 total 10.01 last 70.74) - open/close/unlink 1337 (time 1788791146.69 total 20.01 last 62.87) - open/close/unlink 2195 (time 1788791156.69 total 30.02 last 85.77) - open/close/unlink 3129 (time 1788791166.70 total 40.02 last 93.33) - open/close/unlink 4087 (time 1788791176.70 total 50.03 last 95.77) - open/close/unlink 5026 (time 1788791186.71 total 60.03 last 93.85) - open/close/unlink 5893 (time 1788791196.72 total 70.04 last 86.65) - open/close/unlink 6770 (time 1788791206.72 total 80.04 last 87.67) - open/close/unlink 7633 (time 1788791216.72 total 90.04 last 86.27) - open/close/unlink 8462 (time 1788791226.73 total 100.05 last 82.83) - open/close/unlink 9536 (time 1788791236.74 total 110.06 last 107.32) - open/close/unlink 10000 (time 1788791241.55 total 114.87 last 96.38) - open/close/unlink 10940 (time 1788791251.57 total 124.89 last 93.86) - open/close/unlink 11914 (time 1788791261.57 total 134.89 last 97.35) - open/close/unlink 12792 (time 1788791271.57 total 144.90 last 87.79) - open/close/unlink 13702 (time 1788791281.57 total 154.90 last 90.99) - open/close/unlink 14553 (time 1788791291.58 total 164.90 last 85.10) - open/close/unlink 15264 (time 1788791301.58 total 174.90 last 71.08) - open/close/unlink 16054 (time 1788791311.58 total 184.91 last 78.95) - open/close/unlink 16732 (time 1788791321.59 total 194.91 last 67.77) - open/close/unlink 17489 (time 1788791331.60 total 204.92 last 75.63) - open/close/unlink 18255 (time 1788791341.61 total 214.93 last 76.53) - open/close/unlink 19004 (time 1788791351.61 total 224.94 last 74.84) - open/close/unlink 19731 (time 1788791361.62 total 234.94 last 72.66) - open/close/unlink 20000 (time 1788791364.71 total 238.04 last 86.95) - open/close/unlink 20761 (time 1788791374.72 total 248.05 last 76.02) - open/close/unlink 21514 (time 1788791384.75 total 258.07 last 75.11) - open/close/unlink 22247 (time 1788791394.75 total 268.07 last 73.28) - open/close/unlink 23028 (time 1788791404.75 total 278.07 last 78.10) - open/close/unlink 23844 (time 1788791414.76 total 288.09 last 81.49) - open/close/unlink 24860 (time 1788791424.77 total 298.09 last 101.53) - open/close/unlink 25817 (time 1788791434.78 total 308.10 last 95.65) - open/close/unlink 26706 (time 1788791444.78 total 318.10 last 88.87) - open/close/unlink 27354 (time 1788791454.79 total 328.11 last 64.76) - open/close/unlink 28239 (time 1788791464.79 total 338.11 last 88.48) - open/close/unlink 29122 (time 1788791474.79 total 348.11 last 88.27) - open/close/unlink 30000 (time 1788791484.12 total 357.44 last 94.16) - open/close/unlink 30809 (time 1788791494.12 total 367.44 last 80.90) - open/close/unlink 31673 (time 1788791504.12 total 377.44 last 86.37) - open/close/unlink 32513 (time 1788791514.13 total 387.45 last 83.93) - open/close/unlink 33410 (time 1788791524.14 total 397.46 last 89.63) - open/close/unlink 34294 (time 1788791534.15 total 407.47 last 88.33) - open/close/unlink 35300 (time 1788791544.15 total 417.47 last 100.60) - open/close/unlink 36181 (time 1788791554.15 total 427.48 last 88.03) - open/close/unlink 37194 (time 1788791564.16 total 437.48 last 101.24) - open/close/unlink 38119 (time 1788791574.16 total 447.49 last 92.47) - open/close/unlink 39039 (time 1788791584.17 total 457.49 last 91.97) - open/close/unlink 39810 (time 1788791594.17 total 467.49 last 77.06) - open/close/unlink 40000 (time 1788791596.71 total 470.03 last 74.98) - open/close/unlink 40832 (time 1788791606.72 total 480.04 last 83.09) - open/close/unlink 41620 (time 1788791616.73 total 490.05 last 78.72) - open/close/unlink 42514 (time 1788791626.73 total 500.05 last 89.37) - open/close/unlink 43522 (time 1788791636.74 total 510.06 last 100.76) - open/close/unlink 44567 (time 1788791646.74 total 520.06 last 104.50) - open/close/unlink 45604 (time 1788791656.74 total 530.06 last 103.64) - open/close/unlink 46647 (time 1788791666.74 total 540.07 last 104.27) - open/close/unlink 47225 (time 1788791676.75 total 550.07 last 57.80) - op - open/close/unlink 718 (time 1788791136.70 total 10.00 last 71.80) - open/close/unlink 1340 (time 1788791146.71 total 20.01 last 62.16) - open/close/unlink 2192 (time 1788791156.71 total 30.01 last 85.15) - open/close/unlink 3135 (time 1788791166.71 total 40.02 last 94.28) - open/close/unlink 4096 (time 1788791176.72 total 50.02 last 96.02) - open/close/unlink 5039 (time 1788791186.73 total 60.03 last 94.26) - open/close/unlink 5917 (time 1788791196.73 total 70.03 last 87.77) - open/close/unlink 6794 (time 1788791206.74 total 80.04 last 87.66) - open/close/unlink 7655 (time 1788791216.74 total 90.04 last 86.04) - open/close/unlink 8483 (time 1788791226.75 total 100.05 last 82.72) - open/close/unlink 9566 (time 1788791236.75 total 110.06 last 108.28) - open/close/unlink 10000 (time 1788791241.32 total 114.62 last 95.02) - open/close/unlink 10932 (time 1788791251.33 total 124.63 last 93.15) - open/close/unlink 11896 (time 1788791261.33 total 134.63 last 96.37) - open/close/unlink 12779 (time 1788791271.33 total 144.63 last 88.28) - open/close/unlink 13682 (time 1788791281.34 total 154.64 last 90.27) - open/close/unlink 14531 (time 1788791291.35 total 164.65 last 84.83) - open/close/unlink 15259 (time 1788791301.37 total 174.67 last 72.66) - open/close/unlink 16053 (time 1788791311.37 total 184.67 last 79.37) - open/close/unlink 16726 (time 1788791321.38 total 194.68 last 67.23) - open/close/unlink 17475 (time 1788791331.39 total 204.69 last 74.84) - open/close/unlink 18256 (time 1788791341.40 total 214.70 last 78.02) - open/close/unlink 19006 (time 1788791351.41 total 224.71 last 74.94) - open/close/unlink 19731 (time 1788791361.42 total 234.72 last 72.41) - open/close/unlink 20000 (time 1788791364.50 total 237.80 last 87.35) - open/close/unlink 20764 (time 1788791374.50 total 247.80 last 76.39) - open/close/unlink 21531 (time 1788791384.51 total 257.81 last 76.65) - open/close/unlink 22274 (time 1788791394.52 total 267.82 last 74.18) - open/close/unlink 23052 (time 1788791404.53 total 277.83 last 77.75) - open/close/unlink 23863 (time 1788791414.53 total 287.83 last 81.10) - open/close/unlink 24883 (time 1788791424.53 total 297.83 last 101.98) - open/close/unlink 25849 (time 1788791434.54 total 307.84 last 96.53) - open/close/unlink 26737 (time 1788791444.55 total 317.85 last 88.72) - open/close/unlink 27380 (time 1788791454.55 total 327.85 last 64.28) - open/close/unlink 28255 (time 1788791464.56 total 337.86 last 87.44) - open/close/unlink 29147 (time 1788791474.57 total 347.87 last 89.09) - open/close/unlink 30000 (time 1788791483.54 total 356.84 last 95.12) - open/close/unlink 30811 (time 1788791493.54 total 366.84 last 81.05) - open/close/unlink 31685 (time 1788791503.56 total 376.86 last 87.29) - open/close/unlink 32520 (time 1788791513.56 total 386.86 last 83.49) - open/close/unlink 33419 (time 1788791523.57 total 396.87 last 89.80) - open/close/unlink 34294 (time 1788791533.58 total 406.88 last 87.41) - open/close/unlink 35310 (time 1788791543.58 total 416.88 last 101.56) - open/close/unlink 36173 (time 1788791553.59 total 426.89 last 86.22) - open/close/unlink 37214 (time 1788791563.59 total 436.89 last 104.09) - open/close/unlink 38137 (time 1788791573.60 total 446.90 last 92.24) - open/close/unlink 39069 (time 1788791583.60 total 456.90 last 93.15) - open/close/unlink 39840 (time 1788791593.61 total 466.91 last 77.03) - open/close/unlink 40671 (time 1788791603.62 total 476.92 last 83.06) - open/close/unlink 41409 (time 1788791613.62 total 486.92 last 73.77) - open/close/unlink 42323 (time 1788791623.63 total 496.93 last 91.31) - open/close/unlink 43292 (time 1788791633.64 total 506.94 last 96.83) - open/close/unlink 44337 (time 1788791643.64 total 516.94 last 104.50) - open/close/unlink 45346 (time 1788791653.64 total 526.94 last 100.90) - open/close/unlink 46424 (time 1788791663.64 total 536.94 last 107.75) - open/close/unlink 47135 (time 1788791673.65 total 546.96 last 71.02) - open/close/unlink 47653 (time 1788791683.68 total 556.98 last 51.69) - open/close/unlink 48250 (time 1788791693.68 total 566.98 last 59.65) - open/close/unlink 49025 (time 1788791703.70 total 577.00 last 77.40) - open/close/unlink 50000 (time 1788791713.20 total 586.50 last 102.61) - open/close/unlink 51228 (time 1788791723.21 total 596.51 last 122.72) - open/close/unlink 52459 (time 1788791733.21 total 606.51 last 123.06) - open/close/unlink 53447 (time 1788791743.22 total 616.52 last 98.71) - open/close/unlink 54391 (time 1788791753.22 total 626.52 last 94.39) - open/close/unlink 55171 (time 1788791763.23 total 636.53 last 77.91) - open/close/unlink 56128 (time 1788791773.24 total 646.54 last 95.61) - open/close/unlink 57056 (time 1788791783.25 total 656.55 last 92.72) - open/close/unlink 58031 (time 1788791793.25 total 666.55 last 97.49) - open/close/unlink 58870 (time 1788791803.27 total 676.57 last 83.77) - open/close/unlink 59790 (time 1788791813.27 total 686.57 last 91.94) - open/close/unlink 60000 (time 1788791815.64 total 688.94 last 88.73) - open/close/unlink 60892 (time 1788791825.65 total 698.95 last 89.16) - open/close/unlink 61774 (time 1788791835.65 total 708.95 last 88.14) - open/close/unlink 62581 (time 1788791845.65 total 718.95 last 80.70) - open/close/unlink 63405 (time 1788791855.66 total 728.96 last 82.34) - open/close/unlink 64319 (time 1788791865.66 total 738.96 last 91.36) total: 64768 open/close/unlink in 744.11 seconds: 87.04 ops/second en/close/unlink 47759 (time 1788791685.96 total 559.44 last 52.75) - open/close/unlink 48397 (time 1788791695.97 total 569.45 last 63.75) - open/close/unlink 49229 (time 1788791705.98 total 579.46 last 83.13) - open/close/unlink 50000 (time 1788791713.16 total 586.64 last 107.38) - open/close/unlink 51225 (time 1788791723.16 total 596.65 last 122.41) - open/close/unlink 52463 (time 1788791733.17 total 606.65 last 123.73) - open/close/unlink 53441 (time 1788791743.18 total 616.66 last 97.75) - open/close/unlink 54396 (time 1788791753.19 total 626.67 last 95.38) - open/close/unlink 55173 (time 1788791763.20 total 636.68 last 77.64) - open/close/unlink 56128 (time 1788791773.20 total 646.69 last 95.42) - open/close/unlink 57060 (time 1788791783.21 total 656.69 last 93.17) - open/close/unlink 58030 (time 1788791793.21 total 666.69 last 96.97) - open/close/unlink 58880 (time 1788791803.22 total 676.70 last 84.95) - open/close/unlink 59805 (time 1788791813.23 total 686.71 last 92.41) - open/close/unlink 60000 (time 1788791815.42 total 688.90 last 89.02) - open/close/unlink 60889 (time 1788791825.42 total 698.90 last 88.87) - open/close/unlink 61764 (time 1788791835.43 total 708.91 last 87.41) - open/close/unlink 62577 (time 1788791845.44 total 718.92 last 81.23) - open/close/unlink 63383 (time 1788791855.45 total 728.93 last 80.54) - open/close/unlink 64296 (time 1788791865.45 total 738.94 last 91.22) total: 64768 open/close/unlink in 744.50 seconds: 86.99 ops/second en/close/unlink 47754 (time 1788791686.07 total 559.54 last 52.80) - open/close/unlink 48389 (time 1788791696.07 total 569.54 last 63.50) - open/close/unlink 49217 (time 1788791706.08 total 579.55 last 82.73) - open/close/unlink 50000 (time 1788791713.33 total 586.79 last 108.10) - open/close/unlink 51239 (time 1788791723.33 total 596.79 last 123.88) - open/close/unlink 52478 (time 1788791733.34 total 606.80 last 123.80) - open/close/unlink 53461 (time 1788791743.34 total 616.81 last 98.27) - open/close/unlink 54408 (time 1788791753.34 total 626.81 last 94.69) - open/close/unlink 55193 (time 1788791763.35 total 636.82 last 78.40) - open/close/unlink 56149 (time 1788791773.36 total 646.82 last 95.58) - open/close/unlink 57076 (time 1788791783.36 total 656.83 last 92.65) - open/close/unlink 58033 (time 1788791793.37 total 666.83 last 95.64) - open/close/unlink 58877 (time 1788791803.37 total 676.84 last 84.38) - open/close/unlink 59797 (time 1788791813.38 total 686.85 last 91.92) - open/close/unlink 60000 (time 1788791815.53 total 689.00 last 94.22) - open/close/unlink 60891 (time 1788791825.54 total 699.00 last 89.06) - open/close/unlink 61771 (time 1788791835.54 total 709.01 last 87.98) - open/close/unlink 62586 (time 1788791845.55 total 719.01 last 81.45) - open/close/unlink 63391 (time 1788791855.56 total 729.03 last 80.39) - open/close/unlink 64298 (time 1788791865.57 total 739.03 last 90.63) total: 64768 open/close/unlink in 744.63 seconds: 86.98 ops/second en/close/unlink 47763 (time 1788791686.08 total 559.49 last 52.76) - open/close/unlink 48399 (time 1788791696.09 total 569.50 last 63.53) - open/close/unlink 49222 (time 1788791706.09 total 579.50 last 82.29) - open/close/unlink 50000 (time 1788791713.33 total 586.75 last 107.36) - open/close/unlink 51236 (time 1788791723.34 total 596.75 last 123.53) - open/close/unlink 52476 (time 1788791733.34 total 606.76 last 123.94) - open/close/unlink 53447 (time 1788791743.35 total 616.76 last 97.04) - open/close/unlink 54392 (time 1788791753.36 total 626.77 last 94.46) - open/close/unlink 55179 (time 1788791763.36 total 636.77 last 78.68) - open/close/unlink 56137 (time 1788791773.36 total 646.77 last 95.79) - open/close/unlink 57063 (time 1788791783.36 total 656.78 last 92.57) - open/close/unlink 58026 (time 1788791793.38 total 666.80 last 96.11) - open/close/unlink 58866 (time 1788791803.39 total 676.81 last 83.90) - open/close/unlink 59786 (time 1788791813.40 total 686.81 last 91.94) - open/close/unlink 60000 (time 1788791815.70 total 689.12 last 92.96) - open/close/unlink 60887 (time 1788791825.71 total 699.12 last 88.68) - open/close/unlink 61768 (time 1788791835.71 total 709.13 last 88.02) - open/close/unlink 62571 (time 1788791845.72 total 719.14 last 80.24) - open/close/unlink 63386 (time 1788791855.74 total 729.15 last 81.39) - open/close/unlink 64298 (time 1788791865.74 total 739.15 last 91.18) total: 64768 open/close/unlink in 744.85 seconds: 86.95 ops/second en/close/unlink 47751 (time 1788791686.75 total 560.07 last 52.57) - open/close/unlink 48409 (time 1788791696.77 total 570.09 last 65.70) - open/close/unlink 49253 (time 1788791706.77 total 580.10 last 84.34) - open/close/unlink 50000 (time 1788791713.64 total 586.96 last 108.86) - open/close/unlink 51229 (time 1788791723.64 total 596.96 last 122.86) - open/close/unlink 52469 (time 1788791733.65 total 606.97 last 123.90) - open/close/unlink 53440 (time 1788791743.65 total 616.97 last 97.08) - open/close/unlink 54383 (time 1788791753.66 total 626.98 last 94.21) - open/close/unlink 55160 (time 1788791763.67 total 636.99 last 77.63) - open/close/unlink 56132 (time 1788791773.68 total 647.00 last 97.09) - open/close/unlink 57059 (time 1788791783.68 total 657.00 last 92.67) - open/close/unlink 58019 (time 1788791793.69 total 667.01 last 95.95) - open/close/unlink 58866 (time 1788791803.69 total 677.01 last 84.68) - open/close/unlink 59780 (time 1788791813.70 total 687.02 last 91.30) - open/close/unlink 60000 (time 1788791815.99 total 689.31 last 96.14) - open/close/unlink 60891 (time 1788791826.00 total 699.32 last 89.03) - open/close/unlink 61777 (time 1788791836.00 total 709.33 last 88.54) - open/close/unlink 62582 (time 1788791846.00 total 719.33 last 80.50) - open/close/unlink 63405 (time 1788791856.01 total 729.34 last 82.21) - open/close/unlink 64319 (time 1788791866.02 total 739.34 last 91.35) total: 64768 open/close/unlink in 744.77 seconds: 86.96 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg639-server: cannot open 'lustre-ost2': no such pool pdsh@oleg639-client: oleg639-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg639-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg639-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg639-server stop mds service on oleg639-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg639-server unloading modules via unload_modules_local on: 'oleg639-server' oleg639-server: oleg639-server.virtnet: executing unload_modules_local oleg639-server: modules unloaded. pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-client: ssh exited with exit code 2 pdsh@oleg639-client: oleg639-server: ssh exited with exit code 2