== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 10:28:34 (1780324114) start mds service on oleg352-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg352-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg352-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg352-server: oleg352-server.virtnet: executing set_default_debug -1 all pdsh@oleg352-client: oleg352-server: ssh exited with exit code 1 Started lustre-OST0000 oleg352-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg352-client.virtnet: -o user_xattr,flock 192.168.203.152@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92577 330 92247 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12297 393 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12234 330 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 409 (time 1780324153.97 total 10.03 last 40.78) - open/close/unlink 817 (time 1780324163.98 total 20.04 last 40.76) - open/close/unlink 1259 (time 1780324173.98 total 30.04 last 44.20) - open/close/unlink 1817 (time 1780324184.00 total 40.06 last 55.70) - open/close/unlink 2304 (time 1780324194.02 total 50.07 last 48.62) - open/close/unlink 2798 (time 1780324204.03 total 60.08 last 49.35) - open/close/unlink 3305 (time 1780324214.05 total 70.11 last 50.58) - open/close/unlink 3801 (time 1780324224.06 total 80.11 last 49.57) - open/close/unlink 4252 (time 1780324234.07 total 90.13 last 45.05) - open/close/unlink 4815 (time 1780324244.08 total 100.13 last 56.27) - open/close/unlink 5398 (time 1780324254.09 total 110.14 last 58.23) - open/close/unlink 6052 (time 1780324264.09 total 120.15 last 65.39) - open/close/unlink 6715 (time 1780324274.10 total 130.16 last 66.21) - open/close/unlink 7410 (time 1780324284.11 total 140.17 last 69.44) - open/close/unlink 8145 (time 1780324294.12 total 150.18 last 73.44) - open/close/unlink 8726 (time 1780324304.13 total 160.19 last 58.03) - open/close/unlink 9351 (time 1780324314.13 total 170.19 last 62.49) - open/close/unlink 10000 (time 1780324322.48 total 178.53 last 77.79) - open/close/unlink 10671 (time 1780324332.48 total 188.54 last 67.07) - open/close/unlink 11233 (time 1780324342.50 total 198.55 last 56.12) - open/close/unlink 11750 (time 1780324352.51 total 208.56 last 51.65) - open/close/unlink 12425 (time 1780324362.52 total 218.57 last 67.44) - open/close/unlink 13260 (time 1780324372.52 total 228.58 last 83.43) - open/close/unlink 14011 (time 1780324382.54 total 238.59 last 75.02) - open/close/unlink 14786 (time 1780324392.54 total 248.60 last 77.46) - open/close/unlink 15568 (time 1780324402.54 total 258.60 last 78.19) - open/close/unlink 16279 (time 1780324412.55 total 268.61 last 71.02) - open/close/unlink 17128 (time 1780324422.55 total 278.61 last 84.89) - open/close/unlink 17907 (time 1780324432.56 total 288.62 last 77.82) - open/close/unlink 18691 (time 1780324442.57 total 298.63 last 78.34) - open/close/unlink 19450 (time 1780324452.59 total 308.64 last 75.78) - open/close/unlink 20000 (time 1780324460.05 total 316.11 last 73.70) - open/close/unlink 20595 (time 1780324470.07 total 326.13 last 59.36) - open/close/unlink 21277 (time 1780324480.09 total 336.14 last 68.11) - open/close/unlink 21931 (time 1780324490.09 total 346.15 last 65.37) - open/close/unlink 22592 (time 1780324500.09 total 356.15 last 66.07) - open/close/unlink 23268 (time 1780324510.10 total 366.15 last 67.57) - open/close/unlink 23935 (time 1780324520.11 total 376.17 last 66.62) - open/close/unlink 24554 (time 1780324530.11 total 386.17 last 61.88) - open/close/unlink 25156 (time 1780324540.12 total 396.18 last 60.16) - open/close/unlink 25801 (time 1780324550.13 total 406.18 last 64.46) - open/close/unlink 26425 (time 1780324560.13 total 416.19 last 62.38) - open/close/unlink 27126 (time 1780324570.15 total 426.21 last 69.95) - open/close/unlink 27795 (time 1780324580.15 total 436.21 last 66.89) - open/close/unlink 28383 (time 1780324590.16 total 446.21 last 58.79) - open/close/unlink 29092 (time 1780324600.16 total 456.22 last 70.84) - open/close/unlink 29749 (time 1780324610.18 total 466.23 last 65.61) - open/close/unlink 30000 (time 1780324613.93 total 469.98 last 66.96) - open/close/unlink 30614 (time 1780324623.93 total 479.98 last 61.38) - open/close/unlink 31452 (time 1780324633.94 total 489.99 last 83.75) - open/close/unlink 32190 (time 1780324643.95 total 500.01 last 73.67) - open/close/unlink 32836 (time 1780324653.96 total 510.02 last 64.54) - open/close/unlink 33526 (time 1780324663.97 total 520.02 last 68.96) - open/close/unlink 34320 (time 1780324673.98 total 530.03 last 79.34) - open/close/unlink 35138 (time 1780324683.98 total 540.03 last 81.80) - open/close/unlink 35925 (time 1780324693.98 total 550.03 last 78.69) - open/close/unlink 36825 (time 1780324703.99 total 560.04 last 89.94) - open/close/unlink - open/close/unlink 407 (time 1780324154.16 total 10.01 last 40.68) - open/close/unlink 814 (time 1780324164.16 total 20.01 last 40.69) - open/close/unlink 1266 (time 1780324174.17 total 30.02 last 45.14) - open/close/unlink 1829 (time 1780324184.20 total 40.04 last 56.18) - open/close/unlink 2310 (time 1780324194.21 total 50.06 last 48.04) - open/close/unlink 2811 (time 1780324204.22 total 60.07 last 50.05) - open/close/unlink 3309 (time 1780324214.23 total 70.07 last 49.75) - open/close/unlink 3802 (time 1780324224.23 total 80.08 last 49.30) - open/close/unlink 4250 (time 1780324234.23 total 90.08 last 44.79) - open/close/unlink 4819 (time 1780324244.24 total 100.09 last 56.84) - open/close/unlink 5391 (time 1780324254.25 total 110.10 last 57.13) - open/close/unlink 6038 (time 1780324264.26 total 120.11 last 64.66) - open/close/unlink 6704 (time 1780324274.27 total 130.12 last 66.51) - open/close/unlink 7407 (time 1780324284.28 total 140.12 last 70.27) - open/close/unlink 8141 (time 1780324294.30 total 150.14 last 73.26) - open/close/unlink 8728 (time 1780324304.32 total 160.17 last 58.53) - open/close/unlink 9366 (time 1780324314.33 total 170.17 last 63.79) - open/close/unlink 10000 (time 1780324322.56 total 178.40 last 77.02) - open/close/unlink 10670 (time 1780324332.56 total 188.41 last 66.98) - open/close/unlink 11221 (time 1780324342.57 total 198.41 last 55.07) - open/close/unlink 11738 (time 1780324352.57 total 208.41 last 51.70) - open/close/unlink 12412 (time 1780324362.57 total 218.41 last 67.39) - open/close/unlink 13246 (time 1780324372.58 total 228.42 last 83.33) - open/close/unlink 13995 (time 1780324382.58 total 238.43 last 74.83) - open/close/unlink 14771 (time 1780324392.59 total 248.44 last 77.54) - open/close/unlink 15550 (time 1780324402.60 total 258.45 last 77.81) - open/close/unlink 16264 (time 1780324412.61 total 268.46 last 71.34) - open/close/unlink 17106 (time 1780324422.61 total 278.46 last 84.17) - open/close/unlink 17879 (time 1780324432.63 total 288.48 last 77.17) - open/close/unlink 18669 (time 1780324442.64 total 298.49 last 78.94) - open/close/unlink 19424 (time 1780324452.65 total 308.49 last 75.45) - open/close/unlink 20000 (time 1780324460.49 total 316.33 last 73.47) - open/close/unlink 20592 (time 1780324470.49 total 326.34 last 59.15) - open/close/unlink 21277 (time 1780324480.50 total 336.35 last 68.48) - open/close/unlink 21921 (time 1780324490.50 total 346.35 last 64.35) - open/close/unlink 22586 (time 1780324500.51 total 356.36 last 66.44) - open/close/unlink 23269 (time 1780324510.53 total 366.38 last 68.19) - open/close/unlink 23941 (time 1780324520.53 total 376.38 last 67.19) - open/close/unlink 24557 (time 1780324530.54 total 386.38 last 61.57) - open/close/unlink 25158 (time 1780324540.54 total 396.39 last 60.08) - open/close/unlink 25792 (time 1780324550.55 total 406.40 last 63.34) - open/close/unlink 26431 (time 1780324560.56 total 416.40 last 63.84) - open/close/unlink 27118 (time 1780324570.56 total 426.41 last 68.68) - open/close/unlink 27797 (time 1780324580.57 total 436.41 last 67.86) - open/close/unlink 28395 (time 1780324590.57 total 446.42 last 59.77) - open/close/unlink 29099 (time 1780324600.58 total 456.43 last 70.33) - open/close/unlink 29747 (time 1780324610.58 total 466.43 last 64.80) - open/close/unlink 30000 (time 1780324614.31 total 470.16 last 67.75) - open/close/unlink 30627 (time 1780324624.32 total 480.17 last 62.67) - open/close/unlink 31460 (time 1780324634.33 total 490.18 last 83.21) - open/close/unlink 32203 (time 1780324644.34 total 500.19 last 74.24) - open/close/unlink 32846 (time 1780324654.35 total 510.20 last 64.21) - open/close/unlink 33548 (time 1780324664.36 total 520.21 last 70.14) - open/close/unlink 34324 (time 1780324674.37 total 530.22 last 77.54) - open/close/unlink 35127 (time 1780324684.38 total 540.22 last 80.23) - open/close/unlink 35899 (time 1780324694.38 total 550.23 last 77.16) - open/close/unlink 36808 (time 1780324704.39 total 560.24 last 90.83) - open/close/unlink - open/close/unlink 408 (time 1780324154.08 total 10.05 last 40.61) - open/close/unlink 808 (time 1780324164.08 total 20.05 last 39.97) - open/close/unlink 1255 (time 1780324174.10 total 30.07 last 44.62) - open/close/unlink 1811 (time 1780324184.10 total 40.07 last 55.60) - open/close/unlink 2295 (time 1780324194.12 total 50.09 last 48.29) - open/close/unlink 2798 (time 1780324204.13 total 60.10 last 50.28) - open/close/unlink 3302 (time 1780324214.13 total 70.10 last 50.40) - open/close/unlink 3799 (time 1780324224.14 total 80.11 last 49.66) - open/close/unlink 4252 (time 1780324234.14 total 90.11 last 45.27) - open/close/unlink 4817 (time 1780324244.15 total 100.12 last 56.45) - open/close/unlink 5393 (time 1780324254.16 total 110.13 last 57.54) - open/close/unlink 6038 (time 1780324264.17 total 120.14 last 64.42) - open/close/unlink 6705 (time 1780324274.18 total 130.15 last 66.65) - open/close/unlink 7410 (time 1780324284.19 total 140.16 last 70.46) - open/close/unlink 8145 (time 1780324294.22 total 150.19 last 73.24) - open/close/unlink 8733 (time 1780324304.23 total 160.20 last 58.76) - open/close/unlink 9362 (time 1780324314.23 total 170.20 last 62.88) - open/close/unlink 10000 (time 1780324322.49 total 178.46 last 77.24) - open/close/unlink 10676 (time 1780324332.50 total 188.47 last 67.54) - open/close/unlink 11225 (time 1780324342.52 total 198.49 last 54.81) - open/close/unlink 11737 (time 1780324352.52 total 208.49 last 51.19) - open/close/unlink 12414 (time 1780324362.52 total 218.49 last 67.67) - open/close/unlink 13249 (time 1780324372.53 total 228.50 last 83.42) - open/close/unlink 13998 (time 1780324382.54 total 238.51 last 74.86) - open/close/unlink 14774 (time 1780324392.54 total 248.51 last 77.59) - open/close/unlink 15552 (time 1780324402.54 total 258.51 last 77.79) - open/close/unlink 16260 (time 1780324412.54 total 268.51 last 70.79) - open/close/unlink 17098 (time 1780324422.55 total 278.52 last 83.77) - open/close/unlink 17875 (time 1780324432.56 total 288.53 last 77.63) - open/close/unlink 18663 (time 1780324442.57 total 298.54 last 78.70) - open/close/unlink 19413 (time 1780324452.57 total 308.54 last 75.00) - open/close/unlink 20000 (time 1780324460.54 total 316.51 last 73.67) - open/close/unlink 20588 (time 1780324470.54 total 326.51 last 58.79) - open/close/unlink 21266 (time 1780324480.55 total 336.52 last 67.76) - open/close/unlink 21915 (time 1780324490.55 total 346.52 last 64.84) - open/close/unlink 22580 (time 1780324500.56 total 356.53 last 66.44) - open/close/unlink 23263 (time 1780324510.57 total 366.54 last 68.28) - open/close/unlink 23929 (time 1780324520.58 total 376.55 last 66.54) - open/close/unlink 24553 (time 1780324530.58 total 386.55 last 62.40) - open/close/unlink 25156 (time 1780324540.59 total 396.56 last 60.23) - open/close/unlink 25794 (time 1780324550.59 total 406.56 last 63.79) - open/close/unlink 26423 (time 1780324560.60 total 416.57 last 62.85) - open/close/unlink 27106 (time 1780324570.60 total 426.57 last 68.29) - open/close/unlink 27784 (time 1780324580.60 total 436.57 last 67.76) - open/close/unlink 28375 (time 1780324590.62 total 446.59 last 59.01) - open/close/unlink 29077 (time 1780324600.64 total 456.61 last 70.07) - open/close/unlink 29725 (time 1780324610.64 total 466.61 last 64.75) - open/close/unlink 30000 (time 1780324614.68 total 470.65 last 68.21) - open/close/unlink 30638 (time 1780324624.69 total 480.66 last 63.74) - open/close/unlink 31454 (time 1780324634.70 total 490.67 last 81.49) - open/close/unlink 32198 (time 1780324644.71 total 500.68 last 74.34) - open/close/unlink 32834 (time 1780324654.71 total 510.68 last 63.59) - open/close/unlink 33551 (time 1780324664.71 total 520.68 last 71.67) - open/close/unlink 34327 (time 1780324674.72 total 530.69 last 77.54) - open/close/unlink 35155 (time 1780324684.73 total 540.70 last 82.74) - open/close/unlink 35935 (time 1780324694.74 total 550.70 last 77.95) - open/close/unlink 36845 (time 1780324704.74 total 560.71 last 90.94) - open/close/unlink - open/close/unlink 404 (time 1780324154.30 total 10.02 last 40.33) - open/close/unlink 808 (time 1780324164.33 total 20.05 last 40.28) - open/close/unlink 1256 (time 1780324174.35 total 30.06 last 44.72) - open/close/unlink 1811 (time 1780324184.35 total 40.07 last 55.49) - open/close/unlink 2293 (time 1780324194.36 total 50.08 last 48.15) - open/close/unlink 2784 (time 1780324204.37 total 60.08 last 49.06) - open/close/unlink 3291 (time 1780324214.38 total 70.09 last 50.67) - open/close/unlink 3787 (time 1780324224.38 total 80.09 last 49.59) - open/close/unlink 4233 (time 1780324234.39 total 90.10 last 44.57) - open/close/unlink 4799 (time 1780324244.39 total 100.11 last 56.56) - open/close/unlink 5374 (time 1780324254.40 total 110.11 last 57.45) - open/close/unlink 6031 (time 1780324264.40 total 120.11 last 65.70) - open/close/unlink 6699 (time 1780324274.42 total 130.13 last 66.72) - open/close/unlink 7408 (time 1780324284.42 total 140.13 last 70.89) - open/close/unlink 8128 (time 1780324294.43 total 150.15 last 71.88) - open/close/unlink 8724 (time 1780324304.45 total 160.16 last 59.53) - open/close/unlink 9365 (time 1780324314.45 total 170.16 last 64.07) - open/close/unlink 10000 (time 1780324322.71 total 178.42 last 76.89) - open/close/unlink 10674 (time 1780324332.71 total 188.43 last 67.37) - open/close/unlink 11222 (time 1780324342.72 total 198.43 last 54.78) - open/close/unlink 11731 (time 1780324352.72 total 208.43 last 50.89) - open/close/unlink 12406 (time 1780324362.73 total 218.44 last 67.44) - open/close/unlink 13230 (time 1780324372.73 total 228.44 last 82.39) - open/close/unlink 13985 (time 1780324382.73 total 238.44 last 75.47) - open/close/unlink 14760 (time 1780324392.74 total 248.45 last 77.47) - open/close/unlink 15545 (time 1780324402.74 total 258.46 last 78.44) - open/close/unlink 16243 (time 1780324412.75 total 268.47 last 69.73) - open/close/unlink 17076 (time 1780324422.77 total 278.48 last 83.21) - open/close/unlink 17869 (time 1780324432.77 total 288.48 last 79.25) - open/close/unlink 18648 (time 1780324442.78 total 298.49 last 77.86) - open/close/unlink 19405 (time 1780324452.79 total 308.50 last 75.64) - open/close/unlink 20000 (time 1780324460.91 total 316.62 last 73.24) - open/close/unlink 20589 (time 1780324470.92 total 326.63 last 58.87) - open/close/unlink 21275 (time 1780324480.92 total 336.63 last 68.58) - open/close/unlink 21930 (time 1780324490.92 total 346.64 last 65.46) - open/close/unlink 22579 (time 1780324500.92 total 356.64 last 64.90) - open/close/unlink 23269 (time 1780324510.93 total 366.64 last 68.96) - open/close/unlink 23948 (time 1780324520.94 total 376.66 last 67.80) - open/close/unlink 24564 (time 1780324530.95 total 386.66 last 61.57) - open/close/unlink 25153 (time 1780324540.96 total 396.67 last 58.82) - open/close/unlink 25792 (time 1780324550.98 total 406.69 last 63.80) - open/close/unlink 26442 (time 1780324560.98 total 416.69 last 64.99) - open/close/unlink 27127 (time 1780324570.98 total 426.69 last 68.49) - open/close/unlink 27806 (time 1780324580.98 total 436.69 last 67.89) - open/close/unlink 28394 (time 1780324590.99 total 446.70 last 58.79) - open/close/unlink 29093 (time 1780324601.00 total 456.71 last 69.81) - open/close/unlink 29740 (time 1780324611.00 total 466.71 last 64.67) - open/close/unlink 30000 (time 1780324614.88 total 470.60 last 66.99) - open/close/unlink 30648 (time 1780324624.89 total 480.60 last 64.76) - open/close/unlink 31454 (time 1780324634.89 total 490.60 last 80.57) - open/close/unlink 32208 (time 1780324644.91 total 500.62 last 75.25) - open/close/unlink 32854 (time 1780324654.92 total 510.63 last 64.57) - open/close/unlink 33562 (time 1780324664.93 total 520.64 last 70.72) - open/close/unlink 34331 (time 1780324674.93 total 530.65 last 76.86) - open/close/unlink 35155 (time 1780324684.95 total 540.66 last 82.27) - open/close/unlink 35937 (time 1780324694.96 total 550.67 last 78.12) - open/close/unlink 36849 (time 1780324704.96 total 560.68 last 91.16) - open/close/unlink - open/close/unlink 411 (time 1780324154.17 total 10.00 last 41.09) - open/close/unlink 819 (time 1780324164.22 total 20.05 last 40.61) - open/close/unlink 1260 (time 1780324174.24 total 30.07 last 44.03) - open/close/unlink 1825 (time 1780324184.25 total 40.08 last 56.44) - open/close/unlink 2308 (time 1780324194.26 total 50.09 last 48.24) - open/close/unlink 2806 (time 1780324204.28 total 60.11 last 49.71) - open/close/unlink 3309 (time 1780324214.28 total 70.11 last 50.28) - open/close/unlink 3806 (time 1780324224.30 total 80.13 last 49.62) - open/close/unlink 4256 (time 1780324234.31 total 90.14 last 44.93) - open/close/unlink 4816 (time 1780324244.32 total 100.15 last 55.98) - open/close/unlink 5399 (time 1780324254.33 total 110.16 last 58.25) - open/close/unlink 6047 (time 1780324264.34 total 120.17 last 64.71) - open/close/unlink 6706 (time 1780324274.35 total 130.18 last 65.83) - open/close/unlink 7411 (time 1780324284.35 total 140.19 last 70.47) - open/close/unlink 8143 (time 1780324294.36 total 150.19 last 73.14) - open/close/unlink 8732 (time 1780324304.37 total 160.21 last 58.83) - open/close/unlink 9368 (time 1780324314.37 total 170.21 last 63.60) - open/close/unlink 10000 (time 1780324322.54 total 178.37 last 77.38) - open/close/unlink 10669 (time 1780324332.55 total 188.38 last 66.87) - open/close/unlink 11225 (time 1780324342.55 total 198.38 last 55.58) - open/close/unlink 11734 (time 1780324352.55 total 208.38 last 50.89) - open/close/unlink 12414 (time 1780324362.56 total 218.39 last 67.96) - open/close/unlink 13239 (time 1780324372.56 total 228.39 last 82.49) - open/close/unlink 13989 (time 1780324382.56 total 238.40 last 74.96) - open/close/unlink 14761 (time 1780324392.57 total 248.40 last 77.15) - open/close/unlink 15535 (time 1780324402.58 total 258.41 last 77.30) - open/close/unlink 16236 (time 1780324412.58 total 268.41 last 70.10) - open/close/unlink 17080 (time 1780324422.59 total 278.42 last 84.34) - open/close/unlink 17849 (time 1780324432.60 total 288.43 last 76.81) - open/close/unlink 18630 (time 1780324442.62 total 298.45 last 77.98) - open/close/unlink 19386 (time 1780324452.62 total 308.45 last 75.57) - open/close/unlink 20000 (time 1780324461.00 total 316.83 last 73.30) - open/close/unlink 20587 (time 1780324471.00 total 326.84 last 58.66) - open/close/unlink 21266 (time 1780324481.02 total 336.85 last 67.81) - open/close/unlink 21918 (time 1780324491.03 total 346.86 last 65.13) - open/close/unlink 22572 (time 1780324501.04 total 356.87 last 65.33) - open/close/unlink 23261 (time 1780324511.05 total 366.88 last 68.86) - open/close/unlink 23936 (time 1780324521.05 total 376.88 last 67.47) - open/close/unlink 24545 (time 1780324531.05 total 386.88 last 60.90) - open/close/unlink 25138 (time 1780324541.06 total 396.89 last 59.23) - open/close/unlink 25774 (time 1780324551.09 total 406.92 last 63.45) - open/close/unlink 26417 (time 1780324561.10 total 416.93 last 64.23) - open/close/unlink 27091 (time 1780324571.10 total 426.93 last 67.37) - open/close/unlink 27767 (time 1780324581.10 total 436.93 last 67.59) - open/close/unlink 28360 (time 1780324591.10 total 446.93 last 59.30) - open/close/unlink 29049 (time 1780324601.10 total 456.93 last 68.89) - open/close/unlink 29697 (time 1780324611.12 total 466.95 last 64.71) - open/close/unlink 30000 (time 1780324615.79 total 471.62 last 64.82) - open/close/unlink 30671 (time 1780324625.79 total 481.62 last 67.10) - open/close/unlink 31463 (time 1780324635.80 total 491.63 last 79.13) - open/close/unlink 32204 (time 1780324645.84 total 501.67 last 73.85) - open/close/unlink 32872 (time 1780324655.86 total 511.70 last 66.61) - open/close/unlink 33618 (time 1780324665.87 total 521.70 last 74.56) - open/close/unlink 34353 (time 1780324675.87 total 531.70 last 73.50) - open/close/unlink 35177 (time 1780324685.89 total 541.72 last 82.27) - open/close/unlink 35983 (time 1780324695.89 total 551.73 last 80.53) - open/close/unlink 36904 (time 1780324705.90 total 561.73 last 92.06) - open/close/unlink 37615 (time 1780324713.99 total 570.05 last 78.95) - open/close/unlink 38424 (time 1780324724.00 total 580.06 last 80.81) - open/close/unlink 39235 (time 1780324734.01 total 590.06 last 81.06) - open/close/unlink 40000 (time 1780324743.28 total 599.33 last 82.54) - open/close/unlink 40868 (time 1780324753.28 total 609.33 last 86.78) - open/close/unlink 41711 (time 1780324763.28 total 619.34 last 84.25) - open/close/unlink 42396 (time 1780324773.29 total 629.34 last 68.49) - open/close/unlink 43131 (time 1780324783.29 total 639.34 last 73.49) - open/close/unlink 43938 (time 1780324793.30 total 649.35 last 80.61) - open/close/unlink 44785 (time 1780324803.31 total 659.36 last 84.62) - open/close/unlink 45739 (time 1780324813.31 total 669.37 last 95.37) - open/close/unlink 46603 (time 1780324823.32 total 679.37 last 86.34) - open/close/unlink 47347 (time 1780324833.33 total 689.38 last 74.33) - open/close/unlink 48114 (time 1780324843.34 total 699.39 last 76.62) - open/close/unlink 48811 (time 1780324853.34 total 709.40 last 69.66) - open/close/unlink 49748 (time 1780324863.35 total 719.40 last 93.64) - open/close/unlink 50000 (time 1780324866.05 total 722.11 last 93.24) - open/close/unlink 50906 (time 1780324876.06 total 732.12 last 90.48) - open/close/unlink 51853 (time 1780324886.07 total 742.13 last 94.61) - open/close/unlink 52859 (time 1780324896.08 total 752.13 last 100.58) - open/close/unlink 53870 (time 1780324906.08 total 762.13 last 101.07) - open/close/unlink 54910 (time 1780324916.09 total 772.14 last 103.93) - open/close/unlink 55925 (time 1780324926.09 total 782.15 last 101.43) - open/close/unlink 56959 (time 1780324936.10 total 792.15 last 103.33) - open/close/unlink 57770 (time 1780324946.11 total 802.17 last 81.01) - open/close/unlink 58599 (time 1780324956.12 total 812.17 last 82.85) - open/close/unlink 59564 (time 1780324966.12 total 822.18 last 96.45) - open/close/unlink 60000 (time 1780324970.44 total 826.50 last 100.94) - open/close/unlink 60888 (time 1780324980.45 total 836.51 last 88.72) - open/close/unlink 61733 (time 1780324990.45 total 846.51 last 84.46) - open/close/unlink 62599 (time 1780325000.45 total 856.51 last 86.60) - open/close/unlink 63313 (time 1780325010.46 total 866.52 last 71.37) - open/close/unlink 64172 (time 1780325020.47 total 876.52 last 85.82) total: 64768 open/close/unlink in 883.38 seconds: 73.32 ops/second 37618 (time 1780324714.75 total 570.72 last 77.26) - open/close/unlink 38430 (time 1780324724.75 total 580.72 last 81.19) - open/close/unlink 39245 (time 1780324734.76 total 590.73 last 81.37) - open/close/unlink 40000 (time 1780324744.03 total 600.00 last 81.50) - open/close/unlink 40878 (time 1780324754.03 total 610.00 last 87.79) - open/close/unlink 41693 (time 1780324764.04 total 620.00 last 81.45) - open/close/unlink 42376 (time 1780324774.04 total 630.01 last 68.25) - open/close/unlink 43116 (time 1780324784.05 total 640.02 last 73.98) - open/close/unlink 43928 (time 1780324794.05 total 650.02 last 81.15) - open/close/unlink 44784 (time 1780324804.06 total 660.03 last 85.55) - open/close/unlink 45716 (time 1780324814.06 total 670.03 last 93.19) - open/close/unlink 46586 (time 1780324824.07 total 680.04 last 86.94) - open/close/unlink 47339 (time 1780324834.08 total 690.05 last 75.22) - open/close/unlink 48078 (time 1780324844.08 total 700.05 last 73.87) - open/close/unlink 48799 (time 1780324854.09 total 710.06 last 72.07) - open/close/unlink 49745 (time 1780324864.09 total 720.06 last 94.55) - open/close/unlink 50000 (time 1780324866.92 total 722.89 last 90.10) - open/close/unlink 50907 (time 1780324876.93 total 732.90 last 90.66) - open/close/unlink 51829 (time 1780324886.93 total 742.90 last 92.18) - open/close/unlink 52867 (time 1780324896.94 total 752.91 last 103.73) - open/close/unlink 53898 (time 1780324906.94 total 762.91 last 103.06) - open/close/unlink 54940 (time 1780324916.94 total 772.91 last 104.19) - open/close/unlink 55943 (time 1780324926.95 total 782.92 last 100.23) - open/close/unlink 56957 (time 1780324936.95 total 792.92 last 101.39) - open/close/unlink 57748 (time 1780324946.96 total 802.93 last 78.97) - open/close/unlink 58618 (time 1780324956.97 total 812.94 last 86.91) - open/close/unlink 59583 (time 1780324966.98 total 822.95 last 96.41) - open/close/unlink 60000 (time 1780324971.21 total 827.18 last 98.59) - open/close/unlink 60883 (time 1780324981.22 total 837.19 last 88.22) - open/close/unlink 61745 (time 1780324991.23 total 847.20 last 86.13) - open/close/unlink 62593 (time 1780325001.24 total 857.21 last 84.70) - open/close/unlink 63310 (time 1780325011.24 total 867.21 last 71.69) - open/close/unlink 64153 (time 1780325021.25 total 877.22 last 84.25) total: 64768 open/close/unlink in 884.90 seconds: 73.19 ops/second 37592 (time 1780324714.40 total 570.24 last 78.34) - open/close/unlink 38400 (time 1780324724.40 total 580.25 last 80.74) - open/close/unlink 39217 (time 1780324734.41 total 590.26 last 81.63) - open/close/unlink 40000 (time 1780324743.99 total 599.84 last 81.74) - open/close/unlink 40876 (time 1780324754.00 total 609.84 last 87.56) - open/close/unlink 41696 (time 1780324764.00 total 619.85 last 81.99) - open/close/unlink 42374 (time 1780324774.01 total 629.86 last 67.72) - open/close/unlink 43122 (time 1780324784.02 total 639.87 last 74.70) - open/close/unlink 43934 (time 1780324794.03 total 649.88 last 81.16) - open/close/unlink 44795 (time 1780324804.04 total 659.88 last 86.03) - open/close/unlink 45739 (time 1780324814.04 total 669.89 last 94.35) - open/close/unlink 46606 (time 1780324824.05 total 679.89 last 86.66) - open/close/unlink 47354 (time 1780324834.05 total 689.90 last 74.77) - open/close/unlink 48100 (time 1780324844.06 total 699.91 last 74.55) - open/close/unlink 48816 (time 1780324854.07 total 709.92 last 71.49) - open/close/unlink 49759 (time 1780324864.08 total 719.93 last 94.23) - open/close/unlink 50000 (time 1780324866.76 total 722.61 last 89.88) - open/close/unlink 50906 (time 1780324876.77 total 732.61 last 90.55) - open/close/unlink 51831 (time 1780324886.77 total 742.62 last 92.42) - open/close/unlink 52860 (time 1780324896.78 total 752.63 last 102.85) - open/close/unlink 53887 (time 1780324906.79 total 762.64 last 102.59) - open/close/unlink 54933 (time 1780324916.80 total 772.65 last 104.47) - open/close/unlink 55931 (time 1780324926.81 total 782.66 last 99.73) - open/close/unlink 56955 (time 1780324936.81 total 792.66 last 102.38) - open/close/unlink 57751 (time 1780324946.82 total 802.67 last 79.52) - open/close/unlink 58607 (time 1780324956.82 total 812.67 last 85.58) - open/close/unlink 59570 (time 1780324966.83 total 822.68 last 96.25) - open/close/unlink 60000 (time 1780324971.18 total 827.03 last 98.80) - open/close/unlink 60875 (time 1780324981.19 total 837.04 last 87.38) - open/close/unlink 61732 (time 1780324991.20 total 847.04 last 85.68) - open/close/unlink 62580 (time 1780325001.20 total 857.05 last 84.75) - open/close/unlink 63291 (time 1780325011.21 total 867.05 last 71.08) - open/close/unlink 64139 (time 1780325021.21 total 877.06 last 84.74) total: 64768 open/close/unlink in 884.86 seconds: 73.20 ops/second 37612 (time 1780324714.97 total 570.68 last 76.24) - open/close/unlink 38424 (time 1780324724.99 total 580.70 last 81.08) - open/close/unlink 39246 (time 1780324735.00 total 590.71 last 82.10) - open/close/unlink 40000 (time 1780324744.29 total 600.00 last 81.20) - open/close/unlink 40883 (time 1780324754.29 total 610.00 last 88.25) - open/close/unlink 41688 (time 1780324764.30 total 620.01 last 80.47) - open/close/unlink 42362 (time 1780324774.32 total 630.03 last 67.23) - open/close/unlink 43104 (time 1780324784.33 total 640.04 last 74.14) - open/close/unlink 43923 (time 1780324794.33 total 650.04 last 81.89) - open/close/unlink 44788 (time 1780324804.34 total 660.05 last 86.45) - open/close/unlink 45729 (time 1780324814.35 total 670.06 last 94.01) - open/close/unlink 46603 (time 1780324824.35 total 680.06 last 87.38) - open/close/unlink 47355 (time 1780324834.35 total 690.06 last 75.17) - open/close/unlink 48094 (time 1780324844.36 total 700.07 last 73.84) - open/close/unlink 48817 (time 1780324854.37 total 710.08 last 72.25) - open/close/unlink 49767 (time 1780324864.37 total 720.09 last 94.93) - open/close/unlink 50000 (time 1780324867.02 total 722.73 last 88.14) - open/close/unlink 50888 (time 1780324877.02 total 732.73 last 88.77) - open/close/unlink 51810 (time 1780324887.03 total 742.74 last 92.12) - open/close/unlink 52859 (time 1780324897.03 total 752.74 last 104.87) - open/close/unlink 53893 (time 1780324907.04 total 762.75 last 103.33) - open/close/unlink 54931 (time 1780324917.06 total 772.77 last 103.63) - open/close/unlink 55937 (time 1780324927.07 total 782.78 last 100.45) - open/close/unlink 56955 (time 1780324937.08 total 792.79 last 101.72) - open/close/unlink 57752 (time 1780324947.09 total 802.80 last 79.63) - open/close/unlink 58621 (time 1780324957.09 total 812.80 last 86.85) - open/close/unlink 59578 (time 1780324967.10 total 822.81 last 95.63) - open/close/unlink 60000 (time 1780324971.33 total 827.04 last 99.73) - open/close/unlink 60873 (time 1780324981.33 total 837.05 last 87.28) - open/close/unlink 61737 (time 1780324991.34 total 847.06 last 86.30) - open/close/unlink 62583 (time 1780325001.34 total 857.06 last 84.60) - open/close/unlink 63290 (time 1780325011.35 total 867.06 last 70.70) - open/close/unlink 64125 (time 1780325021.35 total 877.06 last 83.44) total: 64768 open/close/unlink in 885.72 seconds: 73.12 ops/second 37656 (time 1780324715.90 total 571.73 last 75.18) - open/close/unlink 38463 (time 1780324725.90 total 581.73 last 80.70) - open/close/unlink 39282 (time 1780324735.90 total 591.73 last 81.89) - open/close/unlink 40000 (time 1780324744.70 total 600.53 last 81.59) - open/close/unlink 40878 (time 1780324754.71 total 610.54 last 87.74) - open/close/unlink 41674 (time 1780324764.72 total 620.55 last 79.56) - open/close/unlink 42335 (time 1780324774.73 total 630.56 last 66.02) - open/close/unlink 43094 (time 1780324784.73 total 640.56 last 75.90) - open/close/unlink 43911 (time 1780324794.73 total 650.56 last 81.69) - open/close/unlink 44767 (time 1780324804.73 total 660.56 last 85.59) - open/close/unlink 45712 (time 1780324814.73 total 670.56 last 94.50) - open/close/unlink 46581 (time 1780324824.74 total 680.57 last 86.81) - open/close/unlink 47337 (time 1780324834.75 total 690.58 last 75.55) - open/close/unlink 48061 (time 1780324844.75 total 700.58 last 72.39) - open/close/unlink 48797 (time 1780324854.75 total 710.58 last 73.57) - open/close/unlink 49746 (time 1780324864.76 total 720.59 last 94.87) - open/close/unlink 50000 (time 1780324867.74 total 723.57 last 85.19) - open/close/unlink 50931 (time 1780324877.74 total 733.57 last 93.05) - open/close/unlink 51834 (time 1780324887.75 total 743.58 last 90.27) - open/close/unlink 52889 (time 1780324897.75 total 753.58 last 105.48) - open/close/unlink 53931 (time 1780324907.75 total 763.58 last 104.18) - open/close/unlink 54950 (time 1780324917.75 total 773.58 last 101.89) - open/close/unlink 55935 (time 1780324927.76 total 783.59 last 98.39) - open/close/unlink 56967 (time 1780324937.76 total 793.59 last 103.20) - open/close/unlink 57748 (time 1780324947.77 total 803.60 last 78.06) - open/close/unlink 58635 (time 1780324957.77 total 813.60 last 88.64) - open/close/unlink 59609 (time 1780324967.78 total 823.61 last 97.37) - open/close/unlink 60000 (time 1780324971.71 total 827.55 last 99.27) - open/close/unlink 60869 (time 1780324981.72 total 837.55 last 86.86) - open/close/unlink 61736 (time 1780324991.72 total 847.55 last 86.67) - open/close/unlink 62578 (time 1780325001.74 total 857.57 last 84.08) - open/close/unlink 63297 (time 1780325011.74 total 867.57 last 71.87) - open/close/unlink 64125 (time 1780325021.74 total 877.57 last 82.78) total: 64768 open/close/unlink in 886.77 seconds: 73.04 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg352-server: cannot open 'lustre-ost2': no such pool pdsh@oleg352-client: oleg352-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg352-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg352-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg352-server stop mds service on oleg352-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg352-server unloading modules via unload_modules_local on: 'oleg352-server' oleg352-server: oleg352-server.virtnet: executing unload_modules_local oleg352-server: modules unloaded. pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-client: ssh exited with exit code 2 pdsh@oleg352-client: oleg352-server: ssh exited with exit code 2