== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 16:54:50 (1788900890) start mds service on oleg301-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg301-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg301-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg301-server: oleg301-server.virtnet: executing set_default_debug -1 all pdsh@oleg301-client: oleg301-server: ssh exited with exit code 1 Started lustre-OST0000 oleg301-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg301-client.virtnet: -o user_xattr,flock 192.168.203.101@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 657 (time 1788900924.57 total 10.01 last 65.63) - open/close/unlink 1070 (time 1788900934.60 total 20.03 last 41.21) - open/close/unlink 1530 (time 1788900944.60 total 30.03 last 46.00) - open/close/unlink 2272 (time 1788900954.62 total 40.05 last 74.06) - open/close/unlink 3107 (time 1788900964.63 total 50.07 last 83.37) - open/close/unlink 4058 (time 1788900974.63 total 60.07 last 95.09) - open/close/unlink 4940 (time 1788900984.63 total 70.07 last 88.18) - open/close/unlink 5752 (time 1788900994.64 total 80.07 last 81.17) - open/close/unlink 6595 (time 1788901004.66 total 90.09 last 84.14) - open/close/unlink 7341 (time 1788901014.66 total 100.09 last 74.60) - open/close/unlink 7941 (time 1788901024.66 total 110.10 last 59.99) - open/close/unlink 8637 (time 1788901034.66 total 120.10 last 69.59) - open/close/unlink 9406 (time 1788901044.66 total 130.10 last 76.89) - open/close/unlink 10000 (time 1788901051.61 total 137.05 last 85.50) - open/close/unlink 10806 (time 1788901061.61 total 147.05 last 80.60) - open/close/unlink 11565 (time 1788901071.63 total 157.07 last 75.75) - open/close/unlink 12389 (time 1788901081.63 total 167.07 last 82.40) - open/close/unlink 13117 (time 1788901091.64 total 177.07 last 72.76) - open/close/unlink 13788 (time 1788901101.65 total 187.09 last 67.01) - open/close/unlink 14454 (time 1788901111.65 total 197.09 last 66.57) - open/close/unlink 15090 (time 1788901121.66 total 207.09 last 63.59) - open/close/unlink 15442 (time 1788901131.72 total 217.15 last 34.98) - open/close/unlink 15692 (time 1788901141.74 total 227.18 last 24.94) - open/close/unlink 15913 (time 1788901152.10 total 237.54 last 21.33) - open/close/unlink 16190 (time 1788901162.11 total 247.55 last 27.66) - open/close/unlink 16581 (time 1788901172.14 total 257.57 last 39.01) - open/close/unlink 16927 (time 1788901182.15 total 267.58 last 34.57) - open/close/unlink 17409 (time 1788901192.15 total 277.58 last 48.19) - open/close/unlink 17883 (time 1788901202.18 total 287.62 last 47.26) - open/close/unlink 18268 (time 1788901212.19 total 297.63 last 38.45) - open/close/unlink 18642 (time 1788901222.23 total 307.67 last 37.26) - open/close/unlink 19208 (time 1788901232.23 total 317.67 last 56.59) - open/close/unlink 19752 (time 1788901242.25 total 327.68 last 54.33) - open/close/unlink 20000 (time 1788901245.86 total 331.30 last 68.57) - open/close/unlink 20477 (time 1788901255.88 total 341.31 last 47.63) - open/close/unlink 20892 (time 1788901265.88 total 351.32 last 41.47) - open/close/unlink 21356 (time 1788901275.90 total 361.33 last 46.34) - open/close/unlink 21745 (time 1788901285.90 total 371.34 last 38.88) - open/close/unlink 22164 (time 1788901295.91 total 381.35 last 41.86) - open/close/unlink 22554 (time 1788901305.91 total 391.35 last 39.00) - open/close/unlink 22986 (time 1788901315.92 total 401.36 last 43.17) - open/close/unlink 23415 (time 1788901325.94 total 411.37 last 42.82) - open/close/unlink 23819 (time 1788901335.96 total 421.39 last 40.32) - open/close/unlink 24264 (time 1788901345.97 total 431.40 last 44.46) - open/close/unlink 24782 (time 1788901355.97 total 441.40 last 51.79) - open/close/unlink 25238 (time 1788901365.98 total 451.42 last 45.54) - open/close/unlink 25704 (time 1788901375.99 total 461.43 last 46.57) - open/close/unlink 26161 (time 1788901386.00 total 471.44 last 45.65) - open/close/unlink 26629 (time 1788901396.00 total 481.44 last 46.78) - open/close/unlink 27111 (time 1788901406.01 total 491.44 last 48.19) - open/close/unlink 27617 (time 1788901416.02 total 501.45 last 50.55) - open/close/unlink 28160 (time 1788901426.04 total 511.47 last 54.19) - open/close/unlink 28690 (time 1788901436.04 total 521.48 last 52.97) - open/close/unlink 29182 (time 1788901446.05 total 531.48 last 49.18) - open/close/unlink 29728 (time 1788901456.07 total 541.51 last 54.47) - open/close/unlink 30000 (time 1788901461.38 total 546.82 last 51.21) - open/close/unlink 30522 (time 1788901471.39 total 556.83 last 52.14) - open/close/u - open/close/unlink 657 (time 1788900924.50 total 10.02 last 65.59) - open/close/unlink 1061 (time 1788900934.50 total 20.02 last 40.38) - open/close/unlink 1516 (time 1788900944.51 total 30.03 last 45.46) - open/close/unlink 2256 (time 1788900954.52 total 40.04 last 73.95) - open/close/unlink 3100 (time 1788900964.52 total 50.04 last 84.40) - open/close/unlink 4050 (time 1788900974.52 total 60.04 last 94.97) - open/close/unlink 4945 (time 1788900984.54 total 70.06 last 89.34) - open/close/unlink 5745 (time 1788900994.54 total 80.06 last 79.98) - open/close/unlink 6589 (time 1788901004.54 total 90.06 last 84.40) - open/close/unlink 7326 (time 1788901014.55 total 100.07 last 73.63) - open/close/unlink 7931 (time 1788901024.56 total 110.08 last 60.47) - open/close/unlink 8626 (time 1788901034.57 total 120.09 last 69.41) - open/close/unlink 9390 (time 1788901044.57 total 130.09 last 76.36) - open/close/unlink 10000 (time 1788901051.78 total 137.30 last 84.69) - open/close/unlink 10810 (time 1788901061.78 total 147.30 last 80.98) - open/close/unlink 11561 (time 1788901071.78 total 157.30 last 75.07) - open/close/unlink 12379 (time 1788901081.79 total 167.31 last 81.76) - open/close/unlink 13118 (time 1788901091.79 total 177.31 last 73.86) - open/close/unlink 13790 (time 1788901101.80 total 187.32 last 67.15) - open/close/unlink 14457 (time 1788901111.80 total 197.32 last 66.69) - open/close/unlink 15096 (time 1788901121.81 total 207.33 last 63.83) - open/close/unlink 15433 (time 1788901131.82 total 217.34 last 33.68) - open/close/unlink 15684 (time 1788901141.82 total 227.34 last 25.10) - open/close/unlink 15906 (time 1788901152.05 total 237.57 last 21.69) - open/close/unlink 16170 (time 1788901162.08 total 247.60 last 26.35) - open/close/unlink 16555 (time 1788901172.08 total 257.60 last 38.48) - open/close/unlink 16901 (time 1788901182.08 total 267.60 last 34.59) - open/close/unlink 17380 (time 1788901192.09 total 277.61 last 47.87) - open/close/unlink 17846 (time 1788901202.10 total 287.62 last 46.54) - open/close/unlink 18235 (time 1788901212.13 total 297.65 last 38.81) - open/close/unlink 18609 (time 1788901222.13 total 307.65 last 37.38) - open/close/unlink 19172 (time 1788901232.14 total 317.66 last 56.26) - open/close/unlink 19721 (time 1788901242.14 total 327.66 last 54.89) - open/close/unlink 20000 (time 1788901246.18 total 331.70 last 69.03) - open/close/unlink 20471 (time 1788901256.18 total 341.70 last 47.10) - open/close/unlink 20883 (time 1788901266.20 total 351.72 last 41.12) - open/close/unlink 21331 (time 1788901276.20 total 361.72 last 44.79) - open/close/unlink 21723 (time 1788901286.21 total 371.73 last 39.17) - open/close/unlink 22142 (time 1788901296.23 total 381.75 last 41.82) - open/close/unlink 22531 (time 1788901306.23 total 391.75 last 38.89) - open/close/unlink 22957 (time 1788901316.24 total 401.76 last 42.59) - open/close/unlink 23388 (time 1788901326.25 total 411.77 last 43.03) - open/close/unlink 23791 (time 1788901336.27 total 421.79 last 40.24) - open/close/unlink 24231 (time 1788901346.27 total 431.79 last 43.99) - open/close/unlink 24746 (time 1788901356.28 total 441.80 last 51.45) - open/close/unlink 25204 (time 1788901366.29 total 451.81 last 45.78) - open/close/unlink 25665 (time 1788901376.30 total 461.82 last 46.05) - open/close/unlink 26120 (time 1788901386.31 total 471.83 last 45.44) - open/close/unlink 26589 (time 1788901396.32 total 481.84 last 46.86) - open/close/unlink 27077 (time 1788901406.32 total 491.84 last 48.78) - open/close/unlink 27587 (time 1788901416.34 total 501.86 last 50.94) - open/close/unlink 28136 (time 1788901426.36 total 511.88 last 54.79) - open/close/unlink 28667 (time 1788901436.36 total 521.88 last 53.07) - open/close/unlink 29170 (time 1788901446.37 total 531.89 last 50.23) - open/close/unlink 29715 (time 1788901456.38 total 541.90 last 54.46) - open/close/unlink 30000 (time 1788901461.89 total 547.41 last 51.79) - open/close/unlink 30515 (time 1788901471.89 total 557.41 last 51.47) - open/close/u - open/close/unlink 653 (time 1788900924.54 total 10.01 last 65.22) - open/close/unlink 1063 (time 1788900934.54 total 20.02 last 40.99) - open/close/unlink 1524 (time 1788900944.55 total 30.02 last 46.07) - open/close/unlink 2261 (time 1788900954.56 total 40.03 last 73.65) - open/close/unlink 3093 (time 1788900964.57 total 50.04 last 83.09) - open/close/unlink 4041 (time 1788900974.58 total 60.05 last 94.74) - open/close/unlink 4928 (time 1788900984.58 total 70.05 last 88.68) - open/close/unlink 5721 (time 1788900994.58 total 80.05 last 79.28) - open/close/unlink 6561 (time 1788901004.59 total 90.06 last 83.94) - open/close/unlink 7300 (time 1788901014.59 total 100.06 last 73.87) - open/close/unlink 7898 (time 1788901024.59 total 110.07 last 59.79) - open/close/unlink 8581 (time 1788901034.60 total 120.07 last 68.28) - open/close/unlink 9350 (time 1788901044.61 total 130.08 last 76.82) - open/close/unlink 10000 (time 1788901052.23 total 137.70 last 85.28) - open/close/unlink 10808 (time 1788901062.23 total 147.70 last 80.78) - open/close/unlink 11559 (time 1788901072.24 total 157.72 last 75.01) - open/close/unlink 12375 (time 1788901082.25 total 167.73 last 81.52) - open/close/unlink 13117 (time 1788901092.25 total 177.73 last 74.19) - open/close/unlink 13783 (time 1788901102.26 total 187.73 last 66.57) - open/close/unlink 14454 (time 1788901112.26 total 197.73 last 67.09) - open/close/unlink 15087 (time 1788901122.27 total 207.74 last 63.26) - open/close/unlink 15415 (time 1788901132.29 total 217.76 last 32.74) - open/close/unlink 15676 (time 1788901142.31 total 227.78 last 26.04) - open/close/unlink 15888 (time 1788901152.32 total 237.79 last 21.18) - open/close/unlink 16161 (time 1788901162.35 total 247.82 last 27.21) - open/close/unlink 16543 (time 1788901172.36 total 257.84 last 38.15) - open/close/unlink 16893 (time 1788901182.37 total 267.84 last 34.98) - open/close/unlink 17372 (time 1788901192.38 total 277.85 last 47.85) - open/close/unlink 17843 (time 1788901202.40 total 287.87 last 47.03) - open/close/unlink 18231 (time 1788901212.40 total 297.87 last 38.78) - open/close/unlink 18613 (time 1788901222.44 total 307.91 last 38.04) - open/close/unlink 19185 (time 1788901232.44 total 317.92 last 57.19) - open/close/unlink 19734 (time 1788901242.45 total 327.92 last 54.89) - open/close/unlink 20000 (time 1788901246.39 total 331.87 last 67.37) - open/close/unlink 20474 (time 1788901256.42 total 341.89 last 47.29) - open/close/unlink 20886 (time 1788901266.42 total 351.90 last 41.16) - open/close/unlink 21342 (time 1788901276.44 total 361.92 last 45.51) - open/close/unlink 21735 (time 1788901286.46 total 371.93 last 39.24) - open/close/unlink 22147 (time 1788901296.50 total 381.97 last 41.05) - open/close/unlink 22551 (time 1788901306.52 total 391.99 last 40.31) - open/close/unlink 22985 (time 1788901316.53 total 402.00 last 43.37) - open/close/unlink 23413 (time 1788901326.53 total 412.00 last 42.78) - open/close/unlink 23814 (time 1788901336.53 total 422.01 last 40.09) - open/close/unlink 24256 (time 1788901346.54 total 432.01 last 44.19) - open/close/unlink 24774 (time 1788901356.55 total 442.02 last 51.74) - open/close/unlink 25236 (time 1788901366.55 total 452.02 last 46.19) - open/close/unlink 25688 (time 1788901376.56 total 462.03 last 45.17) - open/close/unlink 26145 (time 1788901386.58 total 472.05 last 45.61) - open/close/unlink 26618 (time 1788901396.59 total 482.06 last 47.24) - open/close/unlink 27106 (time 1788901406.62 total 492.09 last 48.66) - open/close/unlink 27617 (time 1788901416.62 total 502.09 last 51.09) - open/close/unlink 28167 (time 1788901426.64 total 512.11 last 54.92) - open/close/unlink 28696 (time 1788901436.66 total 522.13 last 52.80) - open/close/unlink 29188 (time 1788901446.66 total 532.13 last 49.19) - open/close/unlink 29734 (time 1788901456.67 total 542.14 last 54.56) - open/close/unlink 30000 (time 1788901461.93 total 547.40 last 50.54) - open/close/unlink 30517 (time 1788901471.94 total 557.42 last 51.62) - open/close/u - open/close/unlink 651 (time 1788900924.49 total 10.00 last 65.09) - open/close/unlink 1069 (time 1788900934.51 total 20.02 last 41.71) - open/close/unlink 1530 (time 1788900944.52 total 30.03 last 46.08) - open/close/unlink 2271 (time 1788900954.52 total 40.03 last 74.07) - open/close/unlink 3105 (time 1788900964.53 total 50.04 last 83.35) - open/close/unlink 4053 (time 1788900974.53 total 60.04 last 94.74) - open/close/unlink 4935 (time 1788900984.54 total 70.05 last 88.12) - open/close/unlink 5738 (time 1788900994.55 total 80.06 last 80.24) - open/close/unlink 6585 (time 1788901004.56 total 90.07 last 84.59) - open/close/unlink 7327 (time 1788901014.57 total 100.08 last 74.16) - open/close/unlink 7934 (time 1788901024.57 total 110.08 last 60.68) - open/close/unlink 8625 (time 1788901034.58 total 120.09 last 69.03) - open/close/unlink 9392 (time 1788901044.59 total 130.09 last 76.68) - open/close/unlink 10000 (time 1788901051.68 total 137.19 last 85.66) - open/close/unlink 10808 (time 1788901061.69 total 147.20 last 80.76) - open/close/unlink 11563 (time 1788901071.69 total 157.20 last 75.45) - open/close/unlink 12385 (time 1788901081.70 total 167.21 last 82.12) - open/close/unlink 13105 (time 1788901091.71 total 177.22 last 71.95) - open/close/unlink 13770 (time 1788901101.71 total 187.22 last 66.49) - open/close/unlink 14436 (time 1788901111.71 total 197.22 last 66.58) - open/close/unlink 15071 (time 1788901121.72 total 207.23 last 63.48) - open/close/unlink 15410 (time 1788901131.80 total 217.31 last 33.63) - open/close/unlink 15660 (time 1788901141.81 total 227.32 last 24.96) - open/close/unlink 15883 (time 1788901152.08 total 237.59 last 21.73) - open/close/unlink 16161 (time 1788901162.10 total 247.61 last 27.75) - open/close/unlink 16554 (time 1788901172.12 total 257.63 last 39.23) - open/close/unlink 16892 (time 1788901182.13 total 267.64 last 33.76) - open/close/unlink 17375 (time 1788901192.13 total 277.64 last 48.29) - open/close/unlink 17848 (time 1788901202.15 total 287.65 last 47.23) - open/close/unlink 18237 (time 1788901212.17 total 297.67 last 38.82) - open/close/unlink 18614 (time 1788901222.19 total 307.69 last 37.62) - open/close/unlink 19181 (time 1788901232.20 total 317.70 last 56.64) - open/close/unlink 19723 (time 1788901242.20 total 327.71 last 54.19) - open/close/unlink 20000 (time 1788901246.15 total 331.66 last 70.17) - open/close/unlink 20469 (time 1788901256.15 total 341.66 last 46.88) - open/close/unlink 20883 (time 1788901266.16 total 351.67 last 41.35) - open/close/unlink 21330 (time 1788901276.18 total 361.69 last 44.62) - open/close/unlink 21716 (time 1788901286.19 total 371.70 last 38.55) - open/close/unlink 22136 (time 1788901296.20 total 381.71 last 41.97) - open/close/unlink 22532 (time 1788901306.20 total 391.71 last 39.58) - open/close/unlink 22964 (time 1788901316.20 total 401.71 last 43.20) - open/close/unlink 23394 (time 1788901326.22 total 411.73 last 42.92) - open/close/unlink 23795 (time 1788901336.23 total 421.74 last 40.08) - open/close/unlink 24230 (time 1788901346.24 total 431.74 last 43.48) - open/close/unlink 24748 (time 1788901356.24 total 441.75 last 51.75) - open/close/unlink 25201 (time 1788901366.26 total 451.76 last 45.24) - open/close/unlink 25658 (time 1788901376.26 total 461.77 last 45.67) - open/close/unlink 26108 (time 1788901386.28 total 471.79 last 44.91) - open/close/unlink 26574 (time 1788901396.31 total 481.82 last 46.49) - open/close/unlink 27064 (time 1788901406.33 total 491.83 last 48.91) - open/close/unlink 27572 (time 1788901416.34 total 501.85 last 50.73) - open/close/unlink 28121 (time 1788901426.36 total 511.87 last 54.79) - open/close/unlink 28651 (time 1788901436.37 total 521.88 last 52.94) - open/close/unlink 29142 (time 1788901446.38 total 531.89 last 49.05) - open/close/unlink 29692 (time 1788901456.39 total 541.90 last 54.95) - open/close/unlink 30000 (time 1788901462.41 total 547.92 last 51.17) - open/close/unlink 30517 (time 1788901472.43 total 557.93 last 51.60) - open/close/u - open/close/unlink 654 (time 1788900924.54 total 10.00 last 65.39) - open/close/unlink 1057 (time 1788900934.55 total 20.00 last 40.29) - open/close/unlink 1514 (time 1788900944.56 total 30.01 last 45.65) - open/close/unlink 2256 (time 1788900954.56 total 40.02 last 74.18) - open/close/unlink 3092 (time 1788900964.56 total 50.02 last 83.55) - open/close/unlink 4039 (time 1788900974.57 total 60.03 last 94.65) - open/close/unlink 4933 (time 1788900984.57 total 70.03 last 89.37) - open/close/unlink 5734 (time 1788900994.58 total 80.04 last 80.02) - open/close/unlink 6577 (time 1788901004.60 total 90.06 last 84.17) - open/close/unlink 7321 (time 1788901014.60 total 100.06 last 74.37) - open/close/unlink 7919 (time 1788901024.61 total 110.07 last 59.76) - open/close/unlink 8597 (time 1788901034.62 total 120.08 last 67.73) - open/close/unlink 9371 (time 1788901044.62 total 130.08 last 77.40) - open/close/unlink 10000 (time 1788901052.09 total 137.55 last 84.20) - open/close/unlink 10805 (time 1788901062.10 total 147.55 last 80.44) - open/close/unlink 11561 (time 1788901072.10 total 157.56 last 75.55) - open/close/unlink 12378 (time 1788901082.11 total 167.57 last 81.66) - open/close/unlink 13106 (time 1788901092.12 total 177.57 last 72.74) - open/close/unlink 13771 (time 1788901102.13 total 187.59 last 66.40) - open/close/unlink 14435 (time 1788901112.13 total 197.59 last 66.38) - open/close/unlink 15059 (time 1788901122.15 total 207.61 last 62.32) - open/close/unlink 15393 (time 1788901132.15 total 217.61 last 33.39) - open/close/unlink 15649 (time 1788901142.16 total 227.62 last 25.58) - open/close/unlink 15867 (time 1788901152.18 total 237.63 last 21.77) - open/close/unlink 16134 (time 1788901162.18 total 247.64 last 26.68) - open/close/unlink 16522 (time 1788901172.21 total 257.66 last 38.71) - open/close/unlink 16864 (time 1788901182.21 total 267.67 last 34.20) - open/close/unlink 17343 (time 1788901192.21 total 277.67 last 47.87) - open/close/unlink 17815 (time 1788901202.22 total 287.68 last 47.17) - open/close/unlink 18204 (time 1788901212.23 total 297.69 last 38.85) - open/close/unlink 18579 (time 1788901222.25 total 307.71 last 37.42) - open/close/unlink 19153 (time 1788901232.26 total 317.71 last 57.40) - open/close/unlink 19696 (time 1788901242.27 total 327.73 last 54.23) - open/close/unlink 20000 (time 1788901246.86 total 332.32 last 66.18) - open/close/unlink 20477 (time 1788901256.87 total 342.32 last 47.68) - open/close/unlink 20887 (time 1788901266.89 total 352.35 last 40.91) - open/close/unlink 21339 (time 1788901276.91 total 362.37 last 45.10) - open/close/unlink 21724 (time 1788901286.92 total 372.38 last 38.45) - open/close/unlink 22136 (time 1788901296.94 total 382.40 last 41.14) - open/close/unlink 22529 (time 1788901306.96 total 392.41 last 39.23) - open/close/unlink 22957 (time 1788901316.96 total 402.42 last 42.78) - open/close/unlink 23379 (time 1788901326.98 total 412.44 last 42.14) - open/close/unlink 23780 (time 1788901336.99 total 422.45 last 40.04) - open/close/unlink 24227 (time 1788901347.00 total 432.46 last 44.64) - open/close/unlink 24741 (time 1788901357.02 total 442.48 last 51.32) - open/close/unlink 25196 (time 1788901367.03 total 452.49 last 45.46) - open/close/unlink 25652 (time 1788901377.05 total 462.50 last 45.53) - open/close/unlink 26111 (time 1788901387.06 total 472.51 last 45.86) - open/close/unlink 26582 (time 1788901397.07 total 482.53 last 47.04) - open/close/unlink 27073 (time 1788901407.08 total 492.53 last 49.06) - open/close/unlink 27584 (time 1788901417.08 total 502.54 last 51.07) - open/close/unlink 28131 (time 1788901427.08 total 512.54 last 54.70) - open/close/unlink 28671 (time 1788901437.09 total 522.55 last 53.95) - open/close/unlink 29167 (time 1788901447.10 total 532.56 last 49.56) - open/close/unlink 29712 (time 1788901457.11 total 542.57 last 54.44) - open/close/unlink 30000 (time 1788901462.93 total 548.39 last 49.51) - open/close/unlink 30521 (time 1788901472.93 total 558.39 last 52.07) - open/close/unlink 30970 (time 1788901481.91 total 567.43 last 45.44) - open/close/unlink 31466 (time 1788901491.91 total 577.43 last 49.56) - open/close/unlink 31987 (time 1788901501.92 total 587.44 last 52.05) - open/close/unlink 32476 (time 1788901511.94 total 597.46 last 48.83) - open/close/unlink 32955 (time 1788901521.95 total 607.47 last 47.82) - open/close/unlink 33505 (time 1788901531.96 total 617.48 last 54.95) - open/close/unlink 34146 (time 1788901541.97 total 627.49 last 64.07) - open/close/unlink 34878 (time 1788901551.98 total 637.50 last 73.12) - open/close/unlink 35517 (time 1788901561.99 total 647.51 last 63.85) - open/close/unlink 36268 (time 1788901572.00 total 657.52 last 75.02) - open/close/unlink 36829 (time 1788901582.01 total 667.53 last 56.05) - open/close/unlink 37382 (time 1788901592.01 total 677.53 last 55.30) - open/close/unlink 38084 (time 1788901602.01 total 687.53 last 70.19) - open/close/unlink 38773 (time 1788901612.01 total 697.53 last 68.86) - open/close/unlink 39665 (time 1788901622.02 total 707.54 last 89.12) - open/close/unlink 40000 (time 1788901626.19 total 711.71 last 80.30) - open/close/unlink 41041 (time 1788901636.20 total 721.72 last 104.08) - open/close/unlink 41824 (time 1788901646.21 total 731.73 last 78.21) - open/close/unlink 42622 (time 1788901656.22 total 741.74 last 79.74) - open/close/unlink 43570 (time 1788901666.22 total 751.74 last 94.72) - open/close/unlink 44317 (time 1788901676.22 total 761.74 last 74.70) - open/close/unlink 45147 (time 1788901686.23 total 771.75 last 82.98) - open/close/unlink 46170 (time 1788901696.23 total 781.75 last 102.28) - open/close/unlink 47054 (time 1788901706.23 total 791.75 last 88.37) - open/close/unlink 48031 (time 1788901716.24 total 801.76 last 97.64) - open/close/unlink 49010 (time 1788901726.24 total 811.76 last 97.83) - open/close/unlink 49874 (time 1788901736.26 total 821.78 last 86.28) - open/close/unlink 50736 (time 1788901746.27 total 831.79 last 86.12) - open/close/unlink 51581 (time 1788901756.28 total 841.80 last 84.42) - open/close/unlink 52397 (time 1788901766.28 total 851.80 last 81.58) - open/close/unlink 53152 (time 1788901776.28 total 861.80 last 75.49) - open/close/unlink 54061 (time 1788901786.28 total 871.80 last 90.87) - open/close/unlink 54982 (time 1788901796.29 total 881.81 last 92.03) - open/close/unlink 55725 (time 1788901806.30 total 891.82 last 74.26) - open/close/unlink 56650 (time 1788901816.31 total 901.83 last 92.40) - open/close/unlink 57415 (time 1788901826.32 total 911.84 last 76.42) - open/close/unlink 58201 (time 1788901836.33 total 921.85 last 78.54) - open/close/unlink 59031 (time 1788901846.33 total 931.85 last 82.96) - open/close/unlink 59997 (time 1788901856.34 total 941.86 last 96.52) - open/close/unlink 61061 (time 1788901866.34 total 951.86 last 106.36) - open/close/unlink 61955 (time 1788901876.35 total 961.87 last 89.31) - open/close/unlink 62789 (time 1788901886.36 total 971.88 last 83.32) - open/close/unlink 63599 (time 1788901896.37 total 981.89 last 80.94) - open/close/unlink 64454 (time 1788901906.38 total 991.90 last 85.43) total: 64768 open/close/unlink in 995.64 seconds: 65.05 ops/second nlink 30979 (time 1788901482.43 total 567.94 last 46.17) - open/close/unlink 31478 (time 1788901492.45 total 577.95 last 49.84) - open/close/unlink 32005 (time 1788901502.46 total 587.97 last 52.62) - open/close/unlink 32502 (time 1788901512.47 total 597.98 last 49.64) - open/close/unlink 32980 (time 1788901522.48 total 607.99 last 47.75) - open/close/unlink 33537 (time 1788901532.49 total 618.00 last 55.63) - open/close/unlink 34193 (time 1788901542.51 total 628.01 last 65.52) - open/close/unlink 34932 (time 1788901552.51 total 638.02 last 73.86) - open/close/unlink 35578 (time 1788901562.52 total 648.02 last 64.58) - open/close/unlink 36307 (time 1788901572.52 total 658.03 last 72.87) - open/close/unlink 36875 (time 1788901582.53 total 668.04 last 56.75) - open/close/unlink 37439 (time 1788901592.54 total 678.05 last 56.32) - open/close/unlink 38140 (time 1788901602.55 total 688.06 last 70.04) - open/close/unlink 38819 (time 1788901612.56 total 698.06 last 67.87) - open/close/unlink 39727 (time 1788901622.56 total 708.07 last 90.77) - open/close/unlink 40000 (time 1788901626.01 total 711.51 last 79.22) - open/close/unlink 41030 (time 1788901636.01 total 721.52 last 102.99) - open/close/unlink 41812 (time 1788901646.01 total 731.52 last 78.19) - open/close/unlink 42608 (time 1788901656.02 total 741.53 last 79.53) - open/close/unlink 43558 (time 1788901666.04 total 751.55 last 94.79) - open/close/unlink 44306 (time 1788901676.05 total 761.56 last 74.71) - open/close/unlink 45139 (time 1788901686.06 total 771.57 last 83.23) - open/close/unlink 46161 (time 1788901696.06 total 781.57 last 102.17) - open/close/unlink 47049 (time 1788901706.07 total 791.58 last 88.76) - open/close/unlink 48013 (time 1788901716.07 total 801.58 last 96.40) - open/close/unlink 48985 (time 1788901726.07 total 811.58 last 97.18) - open/close/unlink 49854 (time 1788901736.08 total 821.59 last 86.80) - open/close/unlink 50726 (time 1788901746.09 total 831.60 last 87.13) - open/close/unlink 51557 (time 1788901756.09 total 841.60 last 83.06) - open/close/unlink 52370 (time 1788901766.10 total 851.60 last 81.29) - open/close/unlink 53141 (time 1788901776.10 total 861.61 last 77.08) - open/close/unlink 54048 (time 1788901786.10 total 871.61 last 90.70) - open/close/unlink 54962 (time 1788901796.11 total 881.61 last 91.33) - open/close/unlink 55699 (time 1788901806.11 total 891.62 last 73.65) - open/close/unlink 56635 (time 1788901816.13 total 901.64 last 93.46) - open/close/unlink 57393 (time 1788901826.14 total 911.64 last 75.75) - open/close/unlink 58187 (time 1788901836.14 total 921.65 last 79.35) - open/close/unlink 59004 (time 1788901846.15 total 931.65 last 81.66) - open/close/unlink 59990 (time 1788901856.15 total 941.66 last 98.55) - open/close/unlink 61060 (time 1788901866.15 total 951.66 last 106.96) - open/close/unlink 61943 (time 1788901876.16 total 961.66 last 88.29) - open/close/unlink 62788 (time 1788901886.16 total 971.67 last 84.43) - open/close/unlink 63587 (time 1788901896.17 total 981.68 last 79.84) - open/close/unlink 64432 (time 1788901906.18 total 991.69 last 84.45) total: 64768 open/close/unlink in 995.66 seconds: 65.05 ops/second nlink 30977 (time 1788901482.94 total 568.40 last 45.56) - open/close/unlink 31471 (time 1788901492.96 total 578.41 last 49.33) - open/close/unlink 31998 (time 1788901502.97 total 588.43 last 52.63) - open/close/unlink 32503 (time 1788901512.97 total 598.43 last 50.48) - open/close/unlink 32990 (time 1788901522.97 total 608.43 last 48.70) - open/close/unlink 33558 (time 1788901532.99 total 618.44 last 56.74) - open/close/unlink 34208 (time 1788901542.99 total 628.45 last 64.96) - open/close/unlink 34956 (time 1788901553.00 total 638.45 last 74.77) - open/close/unlink 35595 (time 1788901563.01 total 648.47 last 63.79) - open/close/unlink 36325 (time 1788901573.01 total 658.47 last 72.99) - open/close/unlink 36882 (time 1788901583.02 total 668.47 last 55.69) - open/close/unlink 37437 (time 1788901593.02 total 678.48 last 55.49) - open/close/unlink 38149 (time 1788901603.02 total 688.48 last 71.19) - open/close/unlink 38834 (time 1788901613.03 total 698.49 last 68.40) - open/close/unlink 39736 (time 1788901623.04 total 708.50 last 90.16) - open/close/unlink 40000 (time 1788901626.22 total 711.68 last 83.01) - open/close/unlink 41031 (time 1788901636.22 total 721.68 last 103.10) - open/close/unlink 41806 (time 1788901646.23 total 731.69 last 77.42) - open/close/unlink 42606 (time 1788901656.23 total 741.69 last 79.97) - open/close/unlink 43544 (time 1788901666.25 total 751.70 last 93.68) - open/close/unlink 44291 (time 1788901676.25 total 761.70 last 74.70) - open/close/unlink 45131 (time 1788901686.25 total 771.71 last 83.99) - open/close/unlink 46159 (time 1788901696.25 total 781.71 last 102.75) - open/close/unlink 47046 (time 1788901706.26 total 791.71 last 88.66) - open/close/unlink 48013 (time 1788901716.26 total 801.72 last 96.68) - open/close/unlink 48986 (time 1788901726.27 total 811.73 last 97.21) - open/close/unlink 49850 (time 1788901736.28 total 821.74 last 86.32) - open/close/unlink 50716 (time 1788901746.29 total 831.75 last 86.50) - open/close/unlink 51560 (time 1788901756.30 total 841.76 last 84.27) - open/close/unlink 52381 (time 1788901766.31 total 851.77 last 82.06) - open/close/unlink 53138 (time 1788901776.32 total 861.78 last 75.64) - open/close/unlink 54044 (time 1788901786.32 total 871.78 last 90.57) - open/close/unlink 54965 (time 1788901796.32 total 881.78 last 92.10) - open/close/unlink 55698 (time 1788901806.33 total 891.79 last 73.24) - open/close/unlink 56627 (time 1788901816.33 total 901.79 last 92.88) - open/close/unlink 57387 (time 1788901826.35 total 911.81 last 75.88) - open/close/unlink 58176 (time 1788901836.35 total 921.81 last 78.85) - open/close/unlink 59008 (time 1788901846.36 total 931.82 last 83.17) - open/close/unlink 59977 (time 1788901856.36 total 941.82 last 96.89) - open/close/unlink 61045 (time 1788901866.36 total 951.82 last 106.79) - open/close/unlink 61933 (time 1788901876.37 total 961.82 last 88.74) - open/close/unlink 62770 (time 1788901886.37 total 971.83 last 83.63) - open/close/unlink 63572 (time 1788901896.38 total 981.84 last 80.17) - open/close/unlink 64432 (time 1788901906.38 total 991.84 last 85.94) total: 64768 open/close/unlink in 995.90 seconds: 65.03 ops/second nlink 30975 (time 1788901481.40 total 566.84 last 45.27) - open/close/unlink 31465 (time 1788901491.41 total 576.84 last 48.97) - open/close/unlink 31985 (time 1788901501.41 total 586.84 last 52.00) - open/close/unlink 32472 (time 1788901511.41 total 596.85 last 48.67) - open/close/unlink 32959 (time 1788901521.42 total 606.85 last 48.68) - open/close/unlink 33491 (time 1788901531.42 total 616.85 last 53.20) - open/close/unlink 34137 (time 1788901541.44 total 626.87 last 64.47) - open/close/unlink 34847 (time 1788901551.45 total 636.88 last 70.94) - open/close/unlink 35486 (time 1788901561.46 total 646.89 last 63.84) - open/close/unlink 36236 (time 1788901571.47 total 656.90 last 74.91) - open/close/unlink 36802 (time 1788901581.49 total 666.92 last 56.50) - open/close/unlink 37363 (time 1788901591.50 total 676.93 last 56.03) - open/close/unlink 38034 (time 1788901601.50 total 686.94 last 67.06) - open/close/unlink 38727 (time 1788901611.51 total 696.95 last 69.23) - open/close/unlink 39618 (time 1788901621.52 total 706.96 last 89.03) - open/close/unlink 40000 (time 1788901626.31 total 711.75 last 79.69) - open/close/unlink 41032 (time 1788901636.32 total 721.75 last 103.16) - open/close/unlink 41797 (time 1788901646.33 total 731.77 last 76.42) - open/close/unlink 42596 (time 1788901656.33 total 741.77 last 79.89) - open/close/unlink 43526 (time 1788901666.33 total 751.77 last 92.98) - open/close/unlink 44276 (time 1788901676.34 total 761.77 last 74.97) - open/close/unlink 45125 (time 1788901686.35 total 771.78 last 84.83) - open/close/unlink 46159 (time 1788901696.35 total 781.78 last 103.37) - open/close/unlink 47046 (time 1788901706.35 total 791.79 last 88.65) - open/close/unlink 48009 (time 1788901716.36 total 801.79 last 96.28) - open/close/unlink 48967 (time 1788901726.36 total 811.80 last 95.75) - open/close/unlink 49844 (time 1788901736.37 total 821.81 last 87.62) - open/close/unlink 50713 (time 1788901746.39 total 831.82 last 86.75) - open/close/unlink 51557 (time 1788901756.39 total 841.83 last 84.35) - open/close/unlink 52392 (time 1788901766.40 total 851.84 last 83.44) - open/close/unlink 53152 (time 1788901776.41 total 861.84 last 75.97) - open/close/unlink 54058 (time 1788901786.41 total 871.84 last 90.58) - open/close/unlink 54980 (time 1788901796.41 total 881.85 last 92.14) - open/close/unlink 55715 (time 1788901806.42 total 891.85 last 73.47) - open/close/unlink 56641 (time 1788901816.42 total 901.86 last 92.54) - open/close/unlink 57401 (time 1788901826.43 total 911.87 last 75.92) - open/close/unlink 58188 (time 1788901836.44 total 921.87 last 78.68) - open/close/unlink 59021 (time 1788901846.44 total 931.87 last 83.30) - open/close/unlink 59989 (time 1788901856.44 total 941.88 last 96.76) - open/close/unlink 61047 (time 1788901866.45 total 951.88 last 105.77) - open/close/unlink 61932 (time 1788901876.45 total 961.89 last 88.43) - open/close/unlink 62766 (time 1788901886.46 total 971.89 last 83.36) - open/close/unlink 63564 (time 1788901896.47 total 981.91 last 79.72) - open/close/unlink 64420 (time 1788901906.48 total 991.92 last 85.50) total: 64768 open/close/unlink in 996.34 seconds: 65.01 ops/second nlink 30972 (time 1788901481.96 total 567.43 last 45.43) - open/close/unlink 31466 (time 1788901491.97 total 577.44 last 49.34) - open/close/unlink 31990 (time 1788901501.99 total 587.47 last 52.29) - open/close/unlink 32480 (time 1788901512.00 total 597.47 last 48.96) - open/close/unlink 32960 (time 1788901522.02 total 607.49 last 47.94) - open/close/unlink 33513 (time 1788901532.02 total 617.49 last 55.27) - open/close/unlink 34163 (time 1788901542.03 total 627.50 last 64.96) - open/close/unlink 34892 (time 1788901552.03 total 637.50 last 72.90) - open/close/unlink 35532 (time 1788901562.03 total 647.50 last 63.99) - open/close/unlink 36280 (time 1788901572.03 total 657.50 last 74.78) - open/close/unlink 36840 (time 1788901582.03 total 667.51 last 55.98) - open/close/unlink 37401 (time 1788901592.05 total 677.52 last 56.02) - open/close/unlink 38090 (time 1788901602.05 total 687.52 last 68.88) - open/close/unlink 38773 (time 1788901612.06 total 697.53 last 68.23) - open/close/unlink 39672 (time 1788901622.07 total 707.54 last 89.87) - open/close/unlink 40000 (time 1788901626.12 total 711.60 last 80.84) - open/close/unlink 41031 (time 1788901636.13 total 721.60 last 103.04) - open/close/unlink 41804 (time 1788901646.13 total 731.61 last 77.28) - open/close/unlink 42591 (time 1788901656.13 total 741.61 last 78.69) - open/close/unlink 43531 (time 1788901666.14 total 751.61 last 93.96) - open/close/unlink 44275 (time 1788901676.14 total 761.61 last 74.39) - open/close/unlink 45112 (time 1788901686.14 total 771.62 last 83.67) - open/close/unlink 46147 (time 1788901696.15 total 781.63 last 103.41) - open/close/unlink 47030 (time 1788901706.16 total 791.63 last 88.25) - open/close/unlink 48006 (time 1788901716.16 total 801.64 last 97.55) - open/close/unlink 48976 (time 1788901726.17 total 811.64 last 96.98) - open/close/unlink 49837 (time 1788901736.17 total 821.64 last 86.09) - open/close/unlink 50706 (time 1788901746.18 total 831.65 last 86.83) - open/close/unlink 51544 (time 1788901756.19 total 841.66 last 83.69) - open/close/unlink 52359 (time 1788901766.19 total 851.67 last 81.46) - open/close/unlink 53123 (time 1788901776.20 total 861.67 last 76.37) - open/close/unlink 54034 (time 1788901786.20 total 871.68 last 91.05) - open/close/unlink 54945 (time 1788901796.21 total 881.68 last 91.04) - open/close/unlink 55681 (time 1788901806.21 total 891.68 last 73.60) - open/close/unlink 56619 (time 1788901816.22 total 901.69 last 93.71) - open/close/unlink 57378 (time 1788901826.23 total 911.70 last 75.83) - open/close/unlink 58157 (time 1788901836.24 total 921.71 last 77.84) - open/close/unlink 58975 (time 1788901846.24 total 931.72 last 81.74) - open/close/unlink 59945 (time 1788901856.25 total 941.72 last 96.97) - open/close/unlink 61006 (time 1788901866.26 total 951.73 last 105.98) - open/close/unlink 61890 (time 1788901876.26 total 961.73 last 88.37) - open/close/unlink 62732 (time 1788901886.26 total 971.74 last 84.18) - open/close/unlink 63540 (time 1788901896.27 total 981.75 last 80.73) - open/close/unlink 64395 (time 1788901906.28 total 991.75 last 85.46) total: 64768 open/close/unlink in 996.53 seconds: 64.99 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg301-server: cannot open 'lustre-ost2': no such pool pdsh@oleg301-client: oleg301-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg301-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg301-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg301-server stop mds service on oleg301-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg301-server unloading modules via unload_modules_local on: 'oleg301-server' oleg301-server: oleg301-server.virtnet: executing unload_modules_local oleg301-server: kmemleak disabled modules unloaded. pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-client: ssh exited with exit code 2 pdsh@oleg301-client: oleg301-server: ssh exited with exit code 2