== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 18:18:21 (1784413101) start mds service on oleg253-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg253-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg253-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg253-server: oleg253-server.virtnet: executing set_default_debug -1 all pdsh@oleg253-client: oleg253-server: ssh exited with exit code 1 Started lustre-OST0000 oleg253-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg253-client.virtnet: -o user_xattr,flock 192.168.202.153@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1372 (time 1784413129.41 total 10.00 last 137.19) - open/close/unlink 2603 (time 1784413139.41 total 20.00 last 123.09) - open/close/unlink 3923 (time 1784413149.42 total 30.01 last 131.87) - open/close/unlink 5072 (time 1784413159.43 total 40.02 last 114.84) - open/close/unlink 6069 (time 1784413169.43 total 50.02 last 99.63) - open/close/unlink 7209 (time 1784413179.44 total 60.03 last 113.93) - open/close/unlink 8158 (time 1784413189.45 total 70.03 last 94.85) - open/close/unlink 9248 (time 1784413199.45 total 80.04 last 108.95) - open/close/unlink 10000 (time 1784413206.66 total 87.24 last 104.37) - open/close/unlink 10831 (time 1784413216.67 total 97.26 last 82.94) - open/close/unlink 11653 (time 1784413226.68 total 107.27 last 82.15) - open/close/unlink 12598 (time 1784413236.68 total 117.27 last 94.46) - open/close/unlink 13770 (time 1784413246.69 total 127.28 last 117.10) - open/close/unlink 14849 (time 1784413256.70 total 137.29 last 107.82) - open/close/unlink 16122 (time 1784413266.70 total 147.29 last 127.29) - open/close/unlink 17409 (time 1784413276.70 total 157.29 last 128.66) - open/close/unlink 18937 (time 1784413286.71 total 167.30 last 152.77) - open/close/unlink 20000 (time 1784413294.52 total 175.10 last 136.14) - open/close/unlink 21338 (time 1784413304.52 total 185.11 last 133.72) - open/close/unlink 22524 (time 1784413314.52 total 195.11 last 118.56) - open/close/unlink 23945 (time 1784413324.53 total 205.12 last 142.01) - open/close/unlink 25153 (time 1784413334.53 total 215.12 last 120.79) - open/close/unlink 26525 (time 1784413344.54 total 225.12 last 137.15) - open/close/unlink 27920 (time 1784413354.54 total 235.12 last 139.50) - open/close/unlink 29368 (time 1784413364.54 total 245.13 last 144.76) - open/close/unlink 30000 (time 1784413369.47 total 250.06 last 128.12) - open/close/unlink 31310 (time 1784413379.47 total 260.06 last 131.00) - open/close/unlink 32622 (time 1784413389.48 total 270.06 last 131.15) - open/close/unlink 33950 (time 1784413399.48 total 280.07 last 132.78) - open/close/unlink 35348 (time 1784413409.48 total 290.07 last 139.71) - open/close/unlink 36693 (time 1784413419.48 total 300.07 last 134.49) - open/close/unlink 38192 (time 1784413429.49 total 310.08 last 149.79) - open/close/unlink 39704 (time 1784413439.50 total 320.09 last 151.01) - open/close/unlink 40000 (time 1784413441.86 total 322.44 last 125.86) - open/close/unlink 41367 (time 1784413451.86 total 332.44 last 136.69) - open/close/unlink 42877 (time 1784413461.86 total 342.45 last 150.97) - open/close/unlink 44319 (time 1784413471.86 total 352.45 last 144.17) - open/close/unlink 45799 (time 1784413481.86 total 362.45 last 147.98) - open/close/unlink 47392 (time 1784413491.86 total 372.45 last 159.26) - open/close/unlink 48908 (time 1784413501.87 total 382.46 last 151.56) - open/close/unlink 50000 (time 1784413509.00 total 389.59 last 153.06) - open/close/unlink 51342 (time 1784413519.00 total 399.59 last 134.16) - open/close/unlink 52741 (time 1784413529.01 total 409.59 last 139.88) - open/close/unlink 54225 (time 1784413539.01 total 419.60 last 148.33) - open/close/unlink 55726 (time 1784413549.01 total 429.60 last 150.07) - open/close/unlink 57228 (time 1784413559.01 total 439.60 last 150.17) - open/close/unlink 58713 (time 1784413569.01 total 449.60 last 148.49) - open/close/unlink 60000 (time 1784413577.58 total 458.17 last 150.24) - open/close/unlink 61539 (time 1784413587.59 total 468.17 last 153.82) - open/close/unlink 63099 (time 1784413597.59 total 478.18 last 155.90) - open/close/unlink 64478 (time 1784413607.60 total 488.18 last 137.86) total: 64768 open/close/unlink in 490.39 seconds: 132.08 ops/second - open/close/unlink 1381 (time 1784413129.74 total 10.00 last 138.07) - open/close/unlink 2605 (time 1784413139.74 total 20.00 last 122.38) - open/close/unlink 3914 (time 1784413149.74 total 30.01 last 130.87) - open/close/unlink 5075 (time 1784413159.75 total 40.01 last 116.04) - open/close/unlink 6056 (time 1784413169.75 total 50.01 last 98.08) - open/close/unlink 7190 (time 1784413179.76 total 60.02 last 113.28) - open/close/unlink 8166 (time 1784413189.76 total 70.03 last 97.56) - open/close/unlink 9256 (time 1784413199.77 total 80.03 last 108.96) - open/close/unlink 10000 (time 1784413206.97 total 87.24 last 103.24) - open/close/unlink 10822 (time 1784413216.98 total 97.24 last 82.16) - open/close/unlink 11653 (time 1784413227.00 total 107.26 last 82.97) - open/close/unlink 12617 (time 1784413237.00 total 117.26 last 96.39) - open/close/unlink 13784 (time 1784413247.01 total 127.27 last 116.59) - open/close/unlink 14871 (time 1784413257.01 total 137.27 last 108.68) - open/close/unlink 16143 (time 1784413267.01 total 147.27 last 127.20) - open/close/unlink 17443 (time 1784413277.01 total 157.27 last 129.98) - open/close/unlink 18963 (time 1784413287.02 total 167.28 last 151.92) - open/close/unlink 20000 (time 1784413294.66 total 174.93 last 135.56) - open/close/unlink 21347 (time 1784413304.67 total 184.93 last 134.69) - open/close/unlink 22547 (time 1784413314.67 total 194.93 last 119.94) - open/close/unlink 23953 (time 1784413324.67 total 204.93 last 140.58) - open/close/unlink 25165 (time 1784413334.67 total 214.94 last 121.18) - open/close/unlink 26531 (time 1784413344.68 total 224.94 last 136.55) - open/close/unlink 27926 (time 1784413354.68 total 234.95 last 139.42) - open/close/unlink 29381 (time 1784413364.69 total 244.95 last 145.44) - open/close/unlink 30000 (time 1784413369.54 total 249.80 last 127.63) - open/close/unlink 31313 (time 1784413379.54 total 259.80 last 131.27) - open/close/unlink 32627 (time 1784413389.54 total 269.80 last 131.37) - open/close/unlink 33948 (time 1784413399.54 total 279.81 last 132.07) - open/close/unlink 35344 (time 1784413409.55 total 289.81 last 139.57) - open/close/unlink 36705 (time 1784413419.55 total 299.81 last 136.05) - open/close/unlink 38213 (time 1784413429.55 total 309.82 last 150.72) - open/close/unlink 39720 (time 1784413439.56 total 319.82 last 150.67) - open/close/unlink 40000 (time 1784413441.75 total 322.01 last 127.95) - open/close/unlink 41356 (time 1784413451.75 total 332.01 last 135.57) - open/close/unlink 42875 (time 1784413461.75 total 342.01 last 151.85) - open/close/unlink 44320 (time 1784413471.75 total 352.02 last 144.47) - open/close/unlink 45796 (time 1784413481.76 total 362.02 last 147.55) - open/close/unlink 47373 (time 1784413491.76 total 372.02 last 157.66) - open/close/unlink 48892 (time 1784413501.76 total 382.02 last 151.85) - open/close/unlink 50000 (time 1784413509.03 total 389.29 last 152.49) - open/close/unlink 51319 (time 1784413519.03 total 399.29 last 131.87) - open/close/unlink 52713 (time 1784413529.04 total 409.30 last 139.29) - open/close/unlink 54201 (time 1784413539.04 total 419.30 last 148.76) - open/close/unlink 55710 (time 1784413549.04 total 429.30 last 150.86) - open/close/unlink 57195 (time 1784413559.05 total 439.31 last 148.45) - open/close/unlink 58673 (time 1784413569.05 total 449.31 last 147.73) - open/close/unlink 60000 (time 1784413577.86 total 458.12 last 150.62) - open/close/unlink 61526 (time 1784413587.86 total 468.13 last 152.57) - open/close/unlink 63092 (time 1784413597.86 total 478.13 last 156.59) - open/close/unlink 64476 (time 1784413607.89 total 488.15 last 138.07) total: 64768 open/close/unlink in 490.32 seconds: 132.09 ops/second - open/close/unlink 1384 (time 1784413129.70 total 10.01 last 138.28) - open/close/unlink 2608 (time 1784413139.70 total 20.01 last 122.39) - open/close/unlink 3916 (time 1784413149.71 total 30.02 last 130.73) - open/close/unlink 5083 (time 1784413159.72 total 40.03 last 116.56) - open/close/unlink 6066 (time 1784413169.72 total 50.03 last 98.27) - open/close/unlink 7203 (time 1784413179.73 total 60.04 last 113.59) - open/close/unlink 8174 (time 1784413189.73 total 70.04 last 97.07) - open/close/unlink 9259 (time 1784413199.73 total 80.04 last 108.50) - open/close/unlink 10000 (time 1784413206.88 total 87.19 last 103.65) - open/close/unlink 10817 (time 1784413216.89 total 97.20 last 81.62) - open/close/unlink 11643 (time 1784413226.90 total 107.21 last 82.57) - open/close/unlink 12597 (time 1784413236.90 total 117.21 last 95.33) - open/close/unlink 13764 (time 1784413246.90 total 127.21 last 116.70) - open/close/unlink 14856 (time 1784413256.91 total 137.22 last 109.11) - open/close/unlink 16121 (time 1784413266.92 total 147.23 last 126.45) - open/close/unlink 17423 (time 1784413276.92 total 157.23 last 130.16) - open/close/unlink 18947 (time 1784413286.92 total 167.23 last 152.34) - open/close/unlink 20000 (time 1784413294.72 total 175.03 last 135.06) - open/close/unlink 21357 (time 1784413304.72 total 185.03 last 135.67) - open/close/unlink 22548 (time 1784413314.72 total 195.03 last 119.09) - open/close/unlink 23958 (time 1784413324.73 total 205.04 last 140.93) - open/close/unlink 25178 (time 1784413334.73 total 215.04 last 121.96) - open/close/unlink 26548 (time 1784413344.73 total 225.04 last 136.96) - open/close/unlink 27931 (time 1784413354.74 total 235.05 last 138.20) - open/close/unlink 29386 (time 1784413364.74 total 245.05 last 145.46) - open/close/unlink 30000 (time 1784413369.58 total 249.89 last 127.04) - open/close/unlink 31308 (time 1784413379.58 total 259.89 last 130.76) - open/close/unlink 32612 (time 1784413389.59 total 269.90 last 130.30) - open/close/unlink 33937 (time 1784413399.59 total 279.90 last 132.43) - open/close/unlink 35337 (time 1784413409.60 total 289.90 last 139.96) - open/close/unlink 36692 (time 1784413419.60 total 299.91 last 135.44) - open/close/unlink 38186 (time 1784413429.60 total 309.91 last 149.34) - open/close/unlink 39684 (time 1784413439.61 total 319.92 last 149.69) - open/close/unlink 40000 (time 1784413442.02 total 322.33 last 131.05) - open/close/unlink 41357 (time 1784413452.03 total 332.33 last 135.67) - open/close/unlink 42875 (time 1784413462.03 total 342.34 last 151.73) - open/close/unlink 44316 (time 1784413472.04 total 352.34 last 144.01) - open/close/unlink 45805 (time 1784413482.04 total 362.34 last 148.89) - open/close/unlink 47392 (time 1784413492.04 total 372.35 last 158.66) - open/close/unlink 48906 (time 1784413502.04 total 382.35 last 151.39) - open/close/unlink 50000 (time 1784413509.35 total 389.66 last 149.70) - open/close/unlink 51329 (time 1784413519.35 total 399.66 last 132.90) - open/close/unlink 52733 (time 1784413529.35 total 409.66 last 140.38) - open/close/unlink 54221 (time 1784413539.35 total 419.66 last 148.79) - open/close/unlink 55713 (time 1784413549.35 total 429.66 last 149.14) - open/close/unlink 57217 (time 1784413559.36 total 439.67 last 150.31) - open/close/unlink 58685 (time 1784413569.37 total 449.67 last 146.71) - open/close/unlink 60000 (time 1784413578.19 total 458.50 last 148.99) - open/close/unlink 61510 (time 1784413588.21 total 468.51 last 150.79) - open/close/unlink 63067 (time 1784413598.21 total 478.52 last 155.64) - open/close/unlink 64440 (time 1784413608.21 total 488.52 last 137.25) total: 64768 open/close/unlink in 491.41 seconds: 131.80 ops/second - open/close/unlink 1382 (time 1784413129.68 total 10.00 last 138.18) - open/close/unlink 2595 (time 1784413139.68 total 20.00 last 121.28) - open/close/unlink 3897 (time 1784413149.68 total 30.00 last 130.18) - open/close/unlink 5061 (time 1784413159.69 total 40.01 last 116.30) - open/close/unlink 6042 (time 1784413169.70 total 50.02 last 97.98) - open/close/unlink 7161 (time 1784413179.70 total 60.03 last 111.86) - open/close/unlink 8132 (time 1784413189.71 total 70.03 last 97.06) - open/close/unlink 9212 (time 1784413199.71 total 80.04 last 107.95) - open/close/unlink 10000 (time 1784413207.46 total 87.78 last 101.73) - open/close/unlink 10808 (time 1784413217.46 total 97.79 last 80.78) - open/close/unlink 11643 (time 1784413227.46 total 107.79 last 83.49) - open/close/unlink 12624 (time 1784413237.47 total 117.79 last 98.08) - open/close/unlink 13781 (time 1784413247.47 total 127.79 last 115.67) - open/close/unlink 14859 (time 1784413257.48 total 137.80 last 107.69) - open/close/unlink 16155 (time 1784413267.48 total 147.80 last 129.58) - open/close/unlink 17457 (time 1784413277.48 total 157.81 last 130.14) - open/close/unlink 18981 (time 1784413287.49 total 167.81 last 152.37) - open/close/unlink 20000 (time 1784413295.13 total 175.45 last 133.38) - open/close/unlink 21375 (time 1784413305.13 total 185.45 last 137.46) - open/close/unlink 22563 (time 1784413315.13 total 195.46 last 118.75) - open/close/unlink 23993 (time 1784413325.14 total 205.46 last 142.88) - open/close/unlink 25199 (time 1784413335.14 total 215.47 last 120.58) - open/close/unlink 26573 (time 1784413345.15 total 225.47 last 137.33) - open/close/unlink 27953 (time 1784413355.15 total 235.47 last 138.00) - open/close/unlink 29391 (time 1784413365.15 total 245.48 last 143.72) - open/close/unlink 30000 (time 1784413369.88 total 250.21 last 128.74) - open/close/unlink 31306 (time 1784413379.89 total 260.22 last 130.50) - open/close/unlink 32614 (time 1784413389.90 total 270.22 last 130.76) - open/close/unlink 33940 (time 1784413399.90 total 280.23 last 132.48) - open/close/unlink 35337 (time 1784413409.91 total 290.23 last 139.64) - open/close/unlink 36692 (time 1784413419.91 total 300.24 last 135.42) - open/close/unlink 38195 (time 1784413429.92 total 310.24 last 150.26) - open/close/unlink 39672 (time 1784413439.92 total 320.24 last 147.65) - open/close/unlink 40000 (time 1784413442.33 total 322.65 last 136.06) - open/close/unlink 41369 (time 1784413452.33 total 332.65 last 136.90) - open/close/unlink 42876 (time 1784413462.34 total 342.66 last 150.60) - open/close/unlink 44309 (time 1784413472.34 total 352.66 last 143.28) - open/close/unlink 45812 (time 1784413482.34 total 362.66 last 150.28) - open/close/unlink 47407 (time 1784413492.34 total 372.66 last 159.49) - open/close/unlink 48911 (time 1784413502.34 total 382.67 last 150.38) - open/close/unlink 50000 (time 1784413509.68 total 390.00 last 148.43) - open/close/unlink 51342 (time 1784413519.68 total 400.01 last 134.15) - open/close/unlink 52746 (time 1784413529.69 total 410.01 last 140.36) - open/close/unlink 54245 (time 1784413539.69 total 420.01 last 149.86) - open/close/unlink 55724 (time 1784413549.69 total 430.02 last 147.82) - open/close/unlink 57232 (time 1784413559.70 total 440.02 last 150.77) - open/close/unlink 58706 (time 1784413569.70 total 450.02 last 147.37) - open/close/unlink 60000 (time 1784413578.56 total 458.88 last 146.06) - open/close/unlink 61538 (time 1784413588.56 total 468.88 last 153.74) - open/close/unlink 63083 (time 1784413598.56 total 478.89 last 154.46) - open/close/unlink 64436 (time 1784413608.57 total 488.89 last 135.25) total: 64768 open/close/unlink in 491.96 seconds: 131.65 ops/second - open/close/unlink 1382 (time 1784413129.74 total 10.00 last 138.18) - open/close/unlink 2608 (time 1784413139.74 total 20.01 last 122.53) - open/close/unlink 3912 (time 1784413149.75 total 30.01 last 130.34) - open/close/unlink 5069 (time 1784413159.75 total 40.01 last 115.66) - open/close/unlink 6055 (time 1784413169.76 total 50.02 last 98.56) - open/close/unlink 7195 (time 1784413179.76 total 60.02 last 113.96) - open/close/unlink 8169 (time 1784413189.76 total 70.02 last 97.39) - open/close/unlink 9248 (time 1784413199.76 total 80.03 last 107.87) - open/close/unlink 10000 (time 1784413207.12 total 87.38 last 102.28) - open/close/unlink 10813 (time 1784413217.13 total 97.39 last 81.20) - open/close/unlink 11649 (time 1784413227.13 total 107.39 last 83.60) - open/close/unlink 12609 (time 1784413237.13 total 117.39 last 95.97) - open/close/unlink 13771 (time 1784413247.14 total 127.40 last 116.14) - open/close/unlink 14853 (time 1784413257.14 total 137.41 last 108.13) - open/close/unlink 16143 (time 1784413267.15 total 147.41 last 128.91) - open/close/unlink 17435 (time 1784413277.15 total 157.42 last 129.15) - open/close/unlink 18959 (time 1784413287.16 total 167.42 last 152.27) - open/close/unlink 20000 (time 1784413294.95 total 175.21 last 133.71) - open/close/unlink 21370 (time 1784413304.95 total 185.21 last 136.99) - open/close/unlink 22580 (time 1784413314.95 total 195.21 last 120.98) - open/close/unlink 23988 (time 1784413324.95 total 205.22 last 140.73) - open/close/unlink 25209 (time 1784413334.96 total 215.22 last 122.08) - open/close/unlink 26586 (time 1784413344.96 total 225.22 last 137.68) - open/close/unlink 27955 (time 1784413354.96 total 235.22 last 136.85) - open/close/unlink 29404 (time 1784413364.96 total 245.23 last 144.86) - open/close/unlink 30000 (time 1784413369.76 total 250.02 last 124.30) - open/close/unlink 31299 (time 1784413379.76 total 260.02 last 129.86) - open/close/unlink 32591 (time 1784413389.76 total 270.03 last 129.20) - open/close/unlink 33909 (time 1784413399.76 total 280.03 last 131.78) - open/close/unlink 35310 (time 1784413409.77 total 290.03 last 140.02) - open/close/unlink 36669 (time 1784413419.77 total 300.03 last 135.87) - open/close/unlink 38168 (time 1784413429.77 total 310.04 last 149.88) - open/close/unlink 39643 (time 1784413439.77 total 320.04 last 147.49) - open/close/unlink 40000 (time 1784413442.36 total 322.63 last 137.89) - open/close/unlink 41382 (time 1784413452.36 total 332.63 last 138.19) - open/close/unlink 42880 (time 1784413462.36 total 342.63 last 149.79) - open/close/unlink 44302 (time 1784413472.37 total 352.63 last 142.18) - open/close/unlink 45812 (time 1784413482.37 total 362.63 last 150.96) - open/close/unlink 47422 (time 1784413492.38 total 372.64 last 160.91) - open/close/unlink 48927 (time 1784413502.38 total 382.64 last 150.50) - open/close/unlink 50000 (time 1784413509.57 total 389.83 last 149.10) - open/close/unlink 51336 (time 1784413519.58 total 399.84 last 133.54) - open/close/unlink 52740 (time 1784413529.58 total 409.84 last 140.38) - open/close/unlink 54230 (time 1784413539.58 total 419.84 last 148.95) - open/close/unlink 55697 (time 1784413549.58 total 429.85 last 146.67) - open/close/unlink 57198 (time 1784413559.58 total 439.85 last 150.08) - open/close/unlink 58656 (time 1784413569.59 total 449.85 last 145.78) - open/close/unlink 60000 (time 1784413578.75 total 459.01 last 146.62) - open/close/unlink 61527 (time 1784413588.75 total 469.02 last 152.68) - open/close/unlink 63060 (time 1784413598.76 total 479.02 last 153.23) - open/close/unlink 64427 (time 1784413608.76 total 489.02 last 136.66) total: 64768 open/close/unlink in 492.60 seconds: 131.48 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg253-server: cannot open 'lustre-ost2': no such pool pdsh@oleg253-client: oleg253-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg253-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg253-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg253-server stop mds service on oleg253-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg253-server unloading modules via unload_modules_local on: 'oleg253-server' oleg253-server: oleg253-server.virtnet: executing unload_modules_local oleg253-server: modules unloaded. pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-client: ssh exited with exit code 2 pdsh@oleg253-client: oleg253-server: ssh exited with exit code 2