== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 03:54:32 (1781682872) start mds service on oleg243-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg243-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg243-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg243-server: oleg243-server.virtnet: executing set_default_debug -1 all pdsh@oleg243-client: oleg243-server: ssh exited with exit code 1 Started lustre-OST0000 oleg243-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg243-client.virtnet: -o user_xattr,flock 192.168.202.143@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 727 (time 1781682903.51 total 10.00 last 72.69) - open/close/unlink 1595 (time 1781682913.51 total 20.00 last 86.80) - open/close/unlink 2538 (time 1781682923.52 total 30.01 last 94.19) - open/close/unlink 3887 (time 1781682933.52 total 40.01 last 134.88) - open/close/unlink 5358 (time 1781682943.53 total 50.02 last 147.05) - open/close/unlink 6986 (time 1781682953.53 total 60.02 last 162.73) - open/close/unlink 8462 (time 1781682963.54 total 70.03 last 147.53) - open/close/unlink 9964 (time 1781682973.54 total 80.03 last 150.15) - open/close/unlink 11611 (time 1781682983.54 total 90.03 last 164.68) - open/close/unlink 12549 (time 1781682993.54 total 100.03 last 93.78) - open/close/unlink 13373 (time 1781683003.54 total 110.03 last 82.39) - open/close/unlink 14029 (time 1781683013.56 total 120.05 last 65.47) - open/close/unlink 14727 (time 1781683023.57 total 130.06 last 69.76) - open/close/unlink 16103 (time 1781683033.58 total 140.07 last 137.51) - open/close/unlink 17425 (time 1781683043.58 total 150.07 last 132.11) - open/close/unlink 18753 (time 1781683053.58 total 160.07 last 132.79) - open/close/unlink 20000 (time 1781683063.33 total 169.82 last 127.89) - open/close/unlink 21401 (time 1781683073.33 total 179.82 last 140.09) - open/close/unlink 22755 (time 1781683083.34 total 189.83 last 135.30) - open/close/unlink 24073 (time 1781683093.35 total 199.84 last 131.73) - open/close/unlink 25358 (time 1781683103.35 total 209.84 last 128.48) - open/close/unlink 26629 (time 1781683113.35 total 219.84 last 127.09) - open/close/unlink 28039 (time 1781683123.35 total 229.84 last 140.98) - open/close/unlink 29312 (time 1781683133.35 total 239.84 last 127.27) - open/close/unlink 30000 (time 1781683138.86 total 245.35 last 124.99) - open/close/unlink 31394 (time 1781683148.86 total 255.35 last 139.34) - open/close/unlink 32833 (time 1781683158.87 total 265.36 last 143.84) - open/close/unlink 34395 (time 1781683168.87 total 275.36 last 156.09) - open/close/unlink 35920 (time 1781683178.88 total 285.37 last 152.38) - open/close/unlink 37361 (time 1781683188.88 total 295.37 last 144.06) - open/close/unlink 38858 (time 1781683198.89 total 305.38 last 149.62) - open/close/unlink 40000 (time 1781683205.78 total 312.27 last 165.61) - open/close/unlink 41627 (time 1781683215.78 total 322.27 last 162.69) - open/close/unlink 43114 (time 1781683225.79 total 332.28 last 148.69) - open/close/unlink 44650 (time 1781683235.79 total 342.28 last 153.55) - open/close/unlink 46042 (time 1781683245.79 total 352.28 last 139.16) - open/close/unlink 47588 (time 1781683255.80 total 362.29 last 154.52) - open/close/unlink 49055 (time 1781683265.80 total 372.29 last 146.69) - open/close/unlink 50000 (time 1781683272.44 total 378.93 last 142.16) - open/close/unlink 51502 (time 1781683282.45 total 388.94 last 150.13) - open/close/unlink 52897 (time 1781683292.45 total 398.94 last 139.46) - open/close/unlink 54350 (time 1781683302.45 total 408.94 last 145.30) - open/close/unlink 55930 (time 1781683312.45 total 418.94 last 157.96) - open/close/unlink 57568 (time 1781683322.46 total 428.95 last 163.76) - open/close/unlink 59076 (time 1781683332.46 total 438.95 last 150.74) - open/close/unlink 60000 (time 1781683337.69 total 444.18 last 176.78) - open/close/unlink 61740 (time 1781683347.69 total 454.18 last 173.95) - open/close/unlink 63416 (time 1781683357.70 total 464.18 last 167.51) total: 64768 open/close/unlink in 473.70 seconds: 136.73 ops/second - open/close/unlink 733 (time 1781682903.93 total 10.01 last 73.24) - open/close/unlink 1625 (time 1781682913.93 total 20.01 last 89.19) - open/close/unlink 2589 (time 1781682923.94 total 30.01 last 96.37) - open/close/unlink 3938 (time 1781682933.94 total 40.02 last 134.84) - open/close/unlink 5425 (time 1781682943.94 total 50.02 last 148.67) - open/close/unlink 7058 (time 1781682953.95 total 60.02 last 163.25) - open/close/unlink 8517 (time 1781682963.95 total 70.03 last 145.85) - open/close/unlink 10000 (time 1781682973.80 total 79.88 last 150.52) - open/close/unlink 11613 (time 1781682983.80 total 89.88 last 161.27) - open/close/unlink 12538 (time 1781682993.81 total 99.89 last 92.44) - open/close/unlink 13369 (time 1781683003.82 total 109.89 last 83.06) - open/close/unlink 14012 (time 1781683013.82 total 119.89 last 64.29) - open/close/unlink 14721 (time 1781683023.84 total 129.91 last 70.76) - open/close/unlink 16093 (time 1781683033.85 total 139.93 last 137.03) - open/close/unlink 17404 (time 1781683043.85 total 149.93 last 131.06) - open/close/unlink 18721 (time 1781683053.86 total 159.94 last 131.57) - open/close/unlink 20000 (time 1781683063.71 total 169.79 last 129.88) - open/close/unlink 21428 (time 1781683073.71 total 179.79 last 142.77) - open/close/unlink 22776 (time 1781683083.72 total 189.79 last 134.74) - open/close/unlink 24091 (time 1781683093.72 total 199.80 last 131.43) - open/close/unlink 25369 (time 1781683103.72 total 209.80 last 127.78) - open/close/unlink 26662 (time 1781683113.73 total 219.81 last 129.23) - open/close/unlink 28059 (time 1781683123.73 total 229.81 last 139.69) - open/close/unlink 29349 (time 1781683133.73 total 239.81 last 128.95) - open/close/unlink 30000 (time 1781683139.07 total 245.15 last 122.03) - open/close/unlink 31408 (time 1781683149.07 total 255.15 last 140.77) - open/close/unlink 32845 (time 1781683159.08 total 265.15 last 143.65) - open/close/unlink 34382 (time 1781683169.08 total 275.16 last 153.62) - open/close/unlink 35910 (time 1781683179.08 total 285.16 last 152.78) - open/close/unlink 37355 (time 1781683189.09 total 295.16 last 144.44) - open/close/unlink 38827 (time 1781683199.09 total 305.16 last 147.16) - open/close/unlink 40000 (time 1781683206.29 total 312.36 last 162.94) - open/close/unlink 41631 (time 1781683216.29 total 322.36 last 163.10) - open/close/unlink 43107 (time 1781683226.29 total 332.37 last 147.54) - open/close/unlink 44630 (time 1781683236.29 total 342.37 last 152.27) - open/close/unlink 46045 (time 1781683246.29 total 352.37 last 141.48) - open/close/unlink 47579 (time 1781683256.30 total 362.37 last 153.35) - open/close/unlink 49024 (time 1781683266.31 total 372.38 last 144.38) - open/close/unlink 50000 (time 1781683273.25 total 379.32 last 140.60) - open/close/unlink 51527 (time 1781683283.25 total 389.33 last 152.65) - open/close/unlink 52896 (time 1781683293.26 total 399.33 last 136.84) - open/close/unlink 54338 (time 1781683303.26 total 409.33 last 144.16) - open/close/unlink 55973 (time 1781683313.26 total 419.34 last 163.48) - open/close/unlink 57618 (time 1781683323.26 total 429.34 last 164.48) - open/close/unlink 59082 (time 1781683333.26 total 439.34 last 146.37) - open/close/unlink 60000 (time 1781683338.41 total 444.49 last 178.27) - open/close/unlink 61753 (time 1781683348.41 total 454.49 last 175.27) - open/close/unlink 63409 (time 1781683358.42 total 464.49 last 165.56) - open/close/unlink 64700 (time 1781683368.44 total 474.52 last 128.79) total: 64768 open/close/unlink in 476.37 seconds: 135.96 ops/second - open/close/unlink 734 (time 1781682903.92 total 10.00 last 73.37) - open/close/unlink 1631 (time 1781682913.93 total 20.01 last 89.62) - open/close/unlink 2590 (time 1781682923.94 total 30.02 last 95.87) - open/close/unlink 3927 (time 1781682933.94 total 40.02 last 133.63) - open/close/unlink 5402 (time 1781682943.95 total 50.03 last 147.41) - open/close/unlink 7038 (time 1781682953.95 total 60.03 last 163.57) - open/close/unlink 8502 (time 1781682963.95 total 70.03 last 146.35) - open/close/unlink 10000 (time 1781682973.84 total 79.92 last 151.50) - open/close/unlink 11604 (time 1781682983.84 total 89.92 last 160.38) - open/close/unlink 12528 (time 1781682993.84 total 99.92 last 92.39) - open/close/unlink 13354 (time 1781683003.85 total 109.93 last 82.56) - open/close/unlink 13990 (time 1781683013.87 total 119.95 last 63.46) - open/close/unlink 14698 (time 1781683023.87 total 129.95 last 70.78) - open/close/unlink 16068 (time 1781683033.88 total 139.96 last 136.84) - open/close/unlink 17393 (time 1781683043.90 total 149.98 last 132.29) - open/close/unlink 18720 (time 1781683053.92 total 160.00 last 132.49) - open/close/unlink 20000 (time 1781683063.73 total 169.81 last 130.42) - open/close/unlink 21419 (time 1781683073.73 total 179.81 last 141.85) - open/close/unlink 22765 (time 1781683083.74 total 189.82 last 134.57) - open/close/unlink 24056 (time 1781683093.74 total 199.82 last 129.04) - open/close/unlink 25327 (time 1781683103.74 total 209.82 last 127.09) - open/close/unlink 26614 (time 1781683113.74 total 219.82 last 128.69) - open/close/unlink 28019 (time 1781683123.74 total 229.82 last 140.48) - open/close/unlink 29298 (time 1781683133.75 total 239.83 last 127.88) - open/close/unlink 30000 (time 1781683139.41 total 245.49 last 123.84) - open/close/unlink 31417 (time 1781683149.42 total 255.50 last 141.65) - open/close/unlink 32869 (time 1781683159.42 total 265.50 last 145.19) - open/close/unlink 34407 (time 1781683169.42 total 275.50 last 153.75) - open/close/unlink 35912 (time 1781683179.42 total 285.50 last 150.46) - open/close/unlink 37379 (time 1781683189.43 total 295.51 last 146.68) - open/close/unlink 38831 (time 1781683199.43 total 305.51 last 145.13) - open/close/unlink 40000 (time 1781683206.42 total 312.50 last 167.33) - open/close/unlink 41624 (time 1781683216.42 total 322.50 last 162.37) - open/close/unlink 43088 (time 1781683226.42 total 332.50 last 146.38) - open/close/unlink 44617 (time 1781683236.42 total 342.50 last 152.85) - open/close/unlink 46035 (time 1781683246.42 total 352.50 last 141.78) - open/close/unlink 47578 (time 1781683256.43 total 362.51 last 154.27) - open/close/unlink 49034 (time 1781683266.43 total 372.51 last 145.57) - open/close/unlink 50000 (time 1781683273.29 total 379.37 last 140.80) - open/close/unlink 51526 (time 1781683283.29 total 389.37 last 152.52) - open/close/unlink 52903 (time 1781683293.30 total 399.38 last 137.63) - open/close/unlink 54348 (time 1781683303.31 total 409.39 last 144.38) - open/close/unlink 55983 (time 1781683313.31 total 419.39 last 163.49) - open/close/unlink 57626 (time 1781683323.31 total 429.39 last 164.27) - open/close/unlink 59101 (time 1781683333.31 total 439.39 last 147.49) - open/close/unlink 60000 (time 1781683338.42 total 444.50 last 176.12) - open/close/unlink 61753 (time 1781683348.42 total 454.50 last 175.27) - open/close/unlink 63385 (time 1781683358.42 total 464.50 last 163.18) - open/close/unlink 64681 (time 1781683368.44 total 474.52 last 129.32) total: 64768 open/close/unlink in 476.80 seconds: 135.84 ops/second - open/close/unlink 737 (time 1781682903.79 total 10.01 last 73.65) - open/close/unlink 1614 (time 1781682913.81 total 20.03 last 87.52) - open/close/unlink 2576 (time 1781682923.81 total 30.03 last 96.19) - open/close/unlink 3922 (time 1781682933.81 total 40.03 last 134.56) - open/close/unlink 5395 (time 1781682943.81 total 50.03 last 147.30) - open/close/unlink 7018 (time 1781682953.81 total 60.03 last 162.27) - open/close/unlink 8483 (time 1781682963.81 total 70.03 last 146.47) - open/close/unlink 9984 (time 1781682973.82 total 80.04 last 150.09) - open/close/unlink 11585 (time 1781682983.82 total 90.04 last 160.05) - open/close/unlink 12515 (time 1781682993.82 total 100.04 last 92.96) - open/close/unlink 13349 (time 1781683003.83 total 110.05 last 83.33) - open/close/unlink 13985 (time 1781683013.86 total 120.08 last 63.42) - open/close/unlink 14694 (time 1781683023.87 total 130.09 last 70.83) - open/close/unlink 16067 (time 1781683033.88 total 140.10 last 137.15) - open/close/unlink 17388 (time 1781683043.89 total 150.11 last 132.00) - open/close/unlink 18726 (time 1781683053.89 total 160.11 last 133.78) - open/close/unlink 20000 (time 1781683063.81 total 170.03 last 128.36) - open/close/unlink 21417 (time 1781683073.82 total 180.04 last 141.66) - open/close/unlink 22739 (time 1781683083.82 total 190.04 last 132.16) - open/close/unlink 24037 (time 1781683093.82 total 200.04 last 129.79) - open/close/unlink 25310 (time 1781683103.82 total 210.04 last 127.26) - open/close/unlink 26594 (time 1781683113.83 total 220.05 last 128.35) - open/close/unlink 27996 (time 1781683123.83 total 230.05 last 140.11) - open/close/unlink 29267 (time 1781683133.83 total 240.05 last 127.10) - open/close/unlink 30000 (time 1781683139.74 total 245.96 last 124.21) - open/close/unlink 31419 (time 1781683149.74 total 255.96 last 141.86) - open/close/unlink 32885 (time 1781683159.74 total 265.96 last 146.56) - open/close/unlink 34431 (time 1781683169.74 total 275.96 last 154.57) - open/close/unlink 35930 (time 1781683179.74 total 285.96 last 149.88) - open/close/unlink 37416 (time 1781683189.75 total 295.97 last 148.57) - open/close/unlink 38854 (time 1781683199.75 total 305.97 last 143.76) - open/close/unlink 40000 (time 1781683206.62 total 312.84 last 166.75) - open/close/unlink 41625 (time 1781683216.62 total 322.84 last 162.46) - open/close/unlink 43064 (time 1781683226.63 total 332.85 last 143.85) - open/close/unlink 44603 (time 1781683236.63 total 342.85 last 153.82) - open/close/unlink 46029 (time 1781683246.63 total 352.85 last 142.58) - open/close/unlink 47546 (time 1781683256.64 total 362.86 last 151.66) - open/close/unlink 49003 (time 1781683266.64 total 372.86 last 145.65) - open/close/unlink 50000 (time 1781683273.61 total 379.83 last 143.13) - open/close/unlink 51513 (time 1781683283.61 total 389.83 last 151.26) - open/close/unlink 52890 (time 1781683293.61 total 399.83 last 137.68) - open/close/unlink 54311 (time 1781683303.62 total 409.84 last 142.01) - open/close/unlink 55959 (time 1781683313.62 total 419.84 last 164.77) - open/close/unlink 57609 (time 1781683323.62 total 429.84 last 164.94) - open/close/unlink 59113 (time 1781683333.63 total 439.84 last 150.37) - open/close/unlink 60000 (time 1781683338.66 total 444.88 last 176.33) - open/close/unlink 61742 (time 1781683348.66 total 454.88 last 174.19) - open/close/unlink 63389 (time 1781683358.66 total 464.88 last 164.64) - open/close/unlink 64655 (time 1781683368.69 total 474.91 last 126.21) total: 64768 open/close/unlink in 477.86 seconds: 135.54 ops/second - open/close/unlink 737 (time 1781682903.86 total 10.01 last 73.62) - open/close/unlink 1619 (time 1781682913.86 total 20.01 last 88.18) - open/close/unlink 2583 (time 1781682923.87 total 30.02 last 96.31) - open/close/unlink 3931 (time 1781682933.88 total 40.02 last 134.78) - open/close/unlink 5399 (time 1781682943.88 total 50.02 last 146.79) - open/close/unlink 7019 (time 1781682953.88 total 60.03 last 161.93) - open/close/unlink 8480 (time 1781682963.89 total 70.03 last 146.04) - open/close/unlink 9993 (time 1781682973.89 total 80.03 last 151.29) - open/close/unlink 11588 (time 1781682983.89 total 90.04 last 159.43) - open/close/unlink 12511 (time 1781682993.90 total 100.05 last 92.19) - open/close/unlink 13355 (time 1781683003.91 total 110.06 last 84.35) - open/close/unlink 13987 (time 1781683013.92 total 120.07 last 63.13) - open/close/unlink 14698 (time 1781683023.92 total 130.07 last 71.06) - open/close/unlink 16074 (time 1781683033.94 total 140.09 last 137.36) - open/close/unlink 17403 (time 1781683043.96 total 150.10 last 132.69) - open/close/unlink 18732 (time 1781683053.96 total 160.11 last 132.88) - open/close/unlink 20000 (time 1781683063.74 total 169.88 last 129.69) - open/close/unlink 21416 (time 1781683073.74 total 179.88 last 141.57) - open/close/unlink 22759 (time 1781683083.74 total 189.89 last 134.29) - open/close/unlink 24077 (time 1781683093.74 total 199.89 last 131.73) - open/close/unlink 25358 (time 1781683103.75 total 209.90 last 128.03) - open/close/unlink 26644 (time 1781683113.75 total 219.90 last 128.60) - open/close/unlink 28040 (time 1781683123.75 total 229.90 last 139.57) - open/close/unlink 29313 (time 1781683133.76 total 239.90 last 127.26) - open/close/unlink 30000 (time 1781683139.34 total 245.49 last 123.05) - open/close/unlink 31414 (time 1781683149.34 total 255.49 last 141.34) - open/close/unlink 32863 (time 1781683159.35 total 265.50 last 144.83) - open/close/unlink 34390 (time 1781683169.35 total 275.50 last 152.69) - open/close/unlink 35898 (time 1781683179.35 total 285.50 last 150.74) - open/close/unlink 37358 (time 1781683189.35 total 295.50 last 145.98) - open/close/unlink 38813 (time 1781683199.36 total 305.50 last 145.47) - open/close/unlink 40000 (time 1781683206.44 total 312.59 last 167.50) - open/close/unlink 41616 (time 1781683216.45 total 322.59 last 161.54) - open/close/unlink 43076 (time 1781683226.45 total 332.60 last 145.92) - open/close/unlink 44605 (time 1781683236.45 total 342.60 last 152.86) - open/close/unlink 46034 (time 1781683246.46 total 352.60 last 142.89) - open/close/unlink 47545 (time 1781683256.46 total 362.60 last 151.08) - open/close/unlink 48994 (time 1781683266.46 total 372.61 last 144.81) - open/close/unlink 50000 (time 1781683273.58 total 379.73 last 141.28) - open/close/unlink 51527 (time 1781683283.58 total 389.73 last 152.70) - open/close/unlink 52907 (time 1781683293.59 total 399.73 last 137.96) - open/close/unlink 54324 (time 1781683303.59 total 409.74 last 141.66) - open/close/unlink 55970 (time 1781683313.59 total 419.74 last 164.51) - open/close/unlink 57616 (time 1781683323.59 total 429.74 last 164.59) - open/close/unlink 59111 (time 1781683333.60 total 439.74 last 149.47) - open/close/unlink 60000 (time 1781683338.64 total 444.79 last 176.26) - open/close/unlink 61726 (time 1781683348.64 total 454.79 last 172.59) - open/close/unlink 63366 (time 1781683358.65 total 464.79 last 163.92) - open/close/unlink 64636 (time 1781683368.66 total 474.81 last 126.79) total: 64768 open/close/unlink in 478.27 seconds: 135.42 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg243-server: cannot open 'lustre-ost2': no such pool pdsh@oleg243-client: oleg243-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg243-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg243-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg243-server stop mds service on oleg243-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg243-server unloading modules via unload_modules_local on: 'oleg243-server' oleg243-server: oleg243-server.virtnet: executing unload_modules_local oleg243-server: modules unloaded. pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-client: ssh exited with exit code 2 pdsh@oleg243-client: oleg243-server: ssh exited with exit code 2