== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:52:18 (1777593138) start mds service on oleg206-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg206-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid start ost1 service on oleg206-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg206-server: oleg206-server.virtnet: executing set_default_debug -1 all pdsh@oleg206-client: oleg206-server: ssh exited with exit code 1 Started lustre-OST0000 oleg206-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg206-client.virtnet: -o user_xattr,flock 192.168.202.106@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12296 392 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1109 (time 1777593168.42 total 10.01 last 110.83) - open/close/unlink 2251 (time 1777593178.43 total 20.01 last 114.17) - open/close/unlink 3459 (time 1777593188.43 total 30.01 last 120.77) - open/close/unlink 4713 (time 1777593198.43 total 40.02 last 125.33) - open/close/unlink 6142 (time 1777593208.44 total 50.02 last 142.86) - open/close/unlink 7542 (time 1777593218.44 total 60.02 last 139.93) - open/close/unlink 8930 (time 1777593228.44 total 70.03 last 138.79) - open/close/unlink 10000 (time 1777593237.21 total 78.79 last 122.10) - open/close/unlink 11433 (time 1777593247.21 total 88.80 last 143.18) - open/close/unlink 12781 (time 1777593257.21 total 98.80 last 134.78) - open/close/unlink 14087 (time 1777593267.22 total 108.80 last 130.55) - open/close/unlink 15259 (time 1777593277.22 total 118.81 last 117.15) - open/close/unlink 16563 (time 1777593287.22 total 128.81 last 130.37) - open/close/unlink 17892 (time 1777593297.23 total 138.81 last 132.86) - open/close/unlink 19276 (time 1777593307.23 total 148.81 last 138.36) - open/close/unlink 20000 (time 1777593312.22 total 153.80 last 145.25) - open/close/unlink 21492 (time 1777593322.22 total 163.80 last 149.16) - open/close/unlink 23039 (time 1777593332.22 total 173.81 last 154.62) - open/close/unlink 24404 (time 1777593342.23 total 183.81 last 136.47) - open/close/unlink 25765 (time 1777593352.23 total 193.81 last 136.09) - open/close/unlink 27301 (time 1777593362.23 total 203.81 last 153.57) - open/close/unlink 28758 (time 1777593372.23 total 213.81 last 145.67) - open/close/unlink 30000 (time 1777593380.81 total 222.40 last 144.70) - open/close/unlink 31555 (time 1777593390.82 total 232.40 last 155.38) - open/close/unlink 32846 (time 1777593400.82 total 242.41 last 129.06) - open/close/unlink 34269 (time 1777593410.86 total 252.45 last 141.74) - open/close/unlink 35540 (time 1777593420.87 total 262.45 last 127.07) - open/close/unlink 37042 (time 1777593430.87 total 272.46 last 150.11) - open/close/unlink 38592 (time 1777593440.88 total 282.46 last 154.92) - open/close/unlink 40000 (time 1777593450.42 total 292.00 last 147.53) - open/close/unlink 41450 (time 1777593460.43 total 302.01 last 144.93) - open/close/unlink 42947 (time 1777593470.43 total 312.01 last 149.68) - open/close/unlink 44486 (time 1777593480.43 total 322.01 last 153.88) - open/close/unlink 45873 (time 1777593490.43 total 332.01 last 138.68) - open/close/unlink 47487 (time 1777593500.43 total 342.01 last 161.39) - open/close/unlink 49079 (time 1777593510.43 total 352.02 last 159.14) - open/close/unlink 50000 (time 1777593516.00 total 357.59 last 165.43) - open/close/unlink 51653 (time 1777593526.01 total 367.59 last 165.21) - open/close/unlink 53373 (time 1777593536.01 total 377.59 last 171.98) - open/close/unlink 55027 (time 1777593546.01 total 387.59 last 165.37) - open/close/unlink 56597 (time 1777593556.02 total 397.61 last 156.83) - open/close/unlink 58125 (time 1777593566.03 total 407.61 last 152.74) - open/close/unlink 59765 (time 1777593576.03 total 417.61 last 163.99) - open/close/unlink 61465 (time 1777593586.03 total 427.62 last 169.88) - open/close/unlink 63046 (time 1777593596.03 total 437.62 last 158.07) - open/close/unlink 64665 (time 1777593606.04 total 447.63 last 161.79) total: 64768 open/close/unlink in 448.42 seconds: 144.44 ops/second - open/close/unlink 1110 (time 1777593168.41 total 10.00 last 110.98) - open/close/unlink 2262 (time 1777593178.41 total 20.01 last 115.15) - open/close/unlink 3466 (time 1777593188.42 total 30.01 last 120.32) - open/close/unlink 4713 (time 1777593198.42 total 40.01 last 124.67) - open/close/unlink 6164 (time 1777593208.43 total 50.02 last 145.02) - open/close/unlink 7560 (time 1777593218.43 total 60.02 last 139.55) - open/close/unlink 8950 (time 1777593228.44 total 70.03 last 138.88) - open/close/unlink 10000 (time 1777593236.93 total 78.52 last 123.75) - open/close/unlink 11447 (time 1777593246.93 total 88.52 last 144.61) - open/close/unlink 12760 (time 1777593256.94 total 98.53 last 131.25) - open/close/unlink 14071 (time 1777593266.94 total 108.53 last 131.03) - open/close/unlink 15255 (time 1777593276.95 total 118.54 last 118.35) - open/close/unlink 16566 (time 1777593286.95 total 128.54 last 131.01) - open/close/unlink 17889 (time 1777593296.96 total 138.55 last 132.24) - open/close/unlink 19279 (time 1777593306.96 total 148.55 last 138.94) - open/close/unlink 20000 (time 1777593311.95 total 153.54 last 144.41) - open/close/unlink 21496 (time 1777593321.96 total 163.55 last 149.57) - open/close/unlink 23041 (time 1777593331.96 total 173.55 last 154.49) - open/close/unlink 24417 (time 1777593341.96 total 183.55 last 137.53) - open/close/unlink 25773 (time 1777593351.96 total 193.56 last 135.57) - open/close/unlink 27279 (time 1777593361.97 total 203.56 last 150.50) - open/close/unlink 28733 (time 1777593371.97 total 213.56 last 145.40) - open/close/unlink 30000 (time 1777593380.64 total 222.23 last 146.15) - open/close/unlink 31541 (time 1777593390.64 total 232.23 last 154.08) - open/close/unlink 32838 (time 1777593400.64 total 242.24 last 129.67) - open/close/unlink 34247 (time 1777593410.64 total 252.24 last 140.90) - open/close/unlink 35547 (time 1777593420.65 total 262.24 last 129.98) - open/close/unlink 37020 (time 1777593430.65 total 272.24 last 147.29) - open/close/unlink 38559 (time 1777593440.65 total 282.24 last 153.81) - open/close/unlink 40000 (time 1777593450.32 total 291.91 last 149.11) - open/close/unlink 41439 (time 1777593460.32 total 301.91 last 143.89) - open/close/unlink 42941 (time 1777593470.32 total 311.91 last 150.19) - open/close/unlink 44479 (time 1777593480.32 total 321.91 last 153.80) - open/close/unlink 45858 (time 1777593490.32 total 331.91 last 137.87) - open/close/unlink 47472 (time 1777593500.32 total 341.92 last 161.32) - open/close/unlink 49049 (time 1777593510.33 total 351.92 last 157.67) - open/close/unlink 50000 (time 1777593516.03 total 357.62 last 166.78) - open/close/unlink 51663 (time 1777593526.03 total 367.62 last 166.28) - open/close/unlink 53377 (time 1777593536.03 total 377.62 last 171.36) - open/close/unlink 55021 (time 1777593546.04 total 387.63 last 164.35) - open/close/unlink 56576 (time 1777593556.04 total 397.63 last 155.39) - open/close/unlink 58107 (time 1777593566.05 total 407.64 last 152.99) - open/close/unlink 59750 (time 1777593576.05 total 417.64 last 164.28) - open/close/unlink 61455 (time 1777593586.06 total 427.65 last 170.40) - open/close/unlink 63035 (time 1777593596.06 total 437.65 last 158.00) - open/close/unlink 64658 (time 1777593606.06 total 447.65 last 162.21) total: 64768 open/close/unlink in 448.52 seconds: 144.40 ops/second - open/close/unlink 1107 (time 1777593168.41 total 10.00 last 110.69) - open/close/unlink 2264 (time 1777593178.42 total 20.01 last 115.65) - open/close/unlink 3469 (time 1777593188.42 total 30.01 last 120.47) - open/close/unlink 4718 (time 1777593198.42 total 40.01 last 124.88) - open/close/unlink 6151 (time 1777593208.42 total 50.01 last 143.29) - open/close/unlink 7547 (time 1777593218.42 total 60.01 last 139.54) - open/close/unlink 8951 (time 1777593228.42 total 70.01 last 140.39) - open/close/unlink 10000 (time 1777593236.99 total 78.58 last 122.45) - open/close/unlink 11444 (time 1777593247.00 total 88.59 last 144.24) - open/close/unlink 12763 (time 1777593257.01 total 98.60 last 131.84) - open/close/unlink 14048 (time 1777593267.01 total 108.60 last 128.46) - open/close/unlink 15227 (time 1777593277.02 total 118.61 last 117.84) - open/close/unlink 16535 (time 1777593287.02 total 128.61 last 130.73) - open/close/unlink 17864 (time 1777593297.02 total 138.61 last 132.89) - open/close/unlink 19260 (time 1777593307.02 total 148.61 last 139.60) - open/close/unlink 20000 (time 1777593312.06 total 153.65 last 146.82) - open/close/unlink 21492 (time 1777593322.06 total 163.65 last 149.17) - open/close/unlink 23035 (time 1777593332.07 total 173.66 last 154.23) - open/close/unlink 24395 (time 1777593342.07 total 183.66 last 135.95) - open/close/unlink 25760 (time 1777593352.07 total 193.66 last 136.48) - open/close/unlink 27285 (time 1777593362.08 total 203.67 last 152.48) - open/close/unlink 28752 (time 1777593372.08 total 213.67 last 146.69) - open/close/unlink 30000 (time 1777593380.80 total 222.39 last 143.05) - open/close/unlink 31563 (time 1777593390.82 total 232.41 last 156.07) - open/close/unlink 32853 (time 1777593400.83 total 242.42 last 128.81) - open/close/unlink 34268 (time 1777593410.86 total 252.45 last 141.07) - open/close/unlink 35560 (time 1777593420.88 total 262.47 last 128.96) - open/close/unlink 37068 (time 1777593430.89 total 272.48 last 150.72) - open/close/unlink 38603 (time 1777593440.89 total 282.48 last 153.49) - open/close/unlink 40000 (time 1777593450.35 total 291.94 last 147.60) - open/close/unlink 41456 (time 1777593460.35 total 301.94 last 145.54) - open/close/unlink 42936 (time 1777593470.36 total 311.95 last 147.92) - open/close/unlink 44461 (time 1777593480.36 total 321.95 last 152.50) - open/close/unlink 45835 (time 1777593490.36 total 331.95 last 137.35) - open/close/unlink 47441 (time 1777593500.37 total 341.96 last 160.53) - open/close/unlink 49027 (time 1777593510.37 total 351.96 last 158.54) - open/close/unlink 50000 (time 1777593516.14 total 357.73 last 168.74) - open/close/unlink 51667 (time 1777593526.14 total 367.73 last 166.65) - open/close/unlink 53373 (time 1777593536.15 total 377.74 last 170.53) - open/close/unlink 55019 (time 1777593546.15 total 387.74 last 164.58) - open/close/unlink 56580 (time 1777593556.15 total 397.74 last 155.99) - open/close/unlink 58127 (time 1777593566.16 total 407.75 last 154.62) - open/close/unlink 59761 (time 1777593576.16 total 417.75 last 163.31) - open/close/unlink 61447 (time 1777593586.17 total 427.76 last 168.47) - open/close/unlink 63032 (time 1777593596.18 total 437.77 last 158.40) - open/close/unlink 64649 (time 1777593606.19 total 447.78 last 161.51) total: 64768 open/close/unlink in 448.79 seconds: 144.32 ops/second - open/close/unlink 1117 (time 1777593168.18 total 10.01 last 111.56) - open/close/unlink 2275 (time 1777593178.18 total 20.02 last 115.76) - open/close/unlink 3476 (time 1777593188.19 total 30.02 last 120.06) - open/close/unlink 4701 (time 1777593198.19 total 40.02 last 122.46) - open/close/unlink 6134 (time 1777593208.19 total 50.03 last 143.24) - open/close/unlink 7532 (time 1777593218.20 total 60.03 last 139.71) - open/close/unlink 8934 (time 1777593228.21 total 70.04 last 140.12) - open/close/unlink 10000 (time 1777593236.80 total 78.63 last 124.10) - open/close/unlink 11425 (time 1777593246.80 total 88.63 last 142.46) - open/close/unlink 12730 (time 1777593256.80 total 98.63 last 130.49) - open/close/unlink 14039 (time 1777593266.80 total 108.63 last 130.89) - open/close/unlink 15237 (time 1777593276.80 total 118.64 last 119.75) - open/close/unlink 16546 (time 1777593286.82 total 128.65 last 130.73) - open/close/unlink 17856 (time 1777593296.82 total 138.65 last 130.98) - open/close/unlink 19246 (time 1777593306.82 total 148.65 last 138.98) - open/close/unlink 20000 (time 1777593311.96 total 153.79 last 146.72) - open/close/unlink 21486 (time 1777593321.96 total 163.80 last 148.54) - open/close/unlink 23025 (time 1777593331.96 total 173.80 last 153.88) - open/close/unlink 24388 (time 1777593341.97 total 183.80 last 136.23) - open/close/unlink 25754 (time 1777593351.97 total 193.80 last 136.57) - open/close/unlink 27270 (time 1777593361.97 total 203.81 last 151.58) - open/close/unlink 28734 (time 1777593371.97 total 213.81 last 146.38) - open/close/unlink 30000 (time 1777593380.71 total 222.54 last 145.00) - open/close/unlink 31553 (time 1777593390.71 total 232.54 last 155.27) - open/close/unlink 32850 (time 1777593400.72 total 242.55 last 129.58) - open/close/unlink 34260 (time 1777593410.72 total 252.55 last 141.00) - open/close/unlink 35546 (time 1777593420.73 total 262.56 last 128.48) - open/close/unlink 37028 (time 1777593430.73 total 272.56 last 148.20) - open/close/unlink 38562 (time 1777593440.73 total 282.56 last 153.38) - open/close/unlink 40000 (time 1777593450.39 total 292.23 last 148.75) - open/close/unlink 41449 (time 1777593460.40 total 302.23 last 144.87) - open/close/unlink 42919 (time 1777593470.40 total 312.23 last 146.98) - open/close/unlink 44463 (time 1777593480.40 total 322.24 last 154.32) - open/close/unlink 45858 (time 1777593490.41 total 332.24 last 139.46) - open/close/unlink 47469 (time 1777593500.41 total 342.24 last 161.03) - open/close/unlink 49048 (time 1777593510.41 total 352.25 last 157.87) - open/close/unlink 50000 (time 1777593516.11 total 357.95 last 167.01) - open/close/unlink 51661 (time 1777593526.12 total 367.95 last 166.07) - open/close/unlink 53386 (time 1777593536.12 total 377.95 last 172.44) - open/close/unlink 55025 (time 1777593546.12 total 387.95 last 163.89) - open/close/unlink 56584 (time 1777593556.13 total 397.96 last 155.73) - open/close/unlink 58111 (time 1777593566.13 total 407.97 last 152.63) - open/close/unlink 59743 (time 1777593576.14 total 417.97 last 163.11) - open/close/unlink 61438 (time 1777593586.14 total 427.97 last 169.49) - open/close/unlink 63014 (time 1777593596.14 total 437.98 last 157.56) - open/close/unlink 64628 (time 1777593606.15 total 447.98 last 161.27) total: 64768 open/close/unlink in 449.18 seconds: 144.19 ops/second - open/close/unlink 1125 (time 1777593168.43 total 10.00 last 112.50) - open/close/unlink 2282 (time 1777593178.43 total 20.00 last 115.68) - open/close/unlink 3477 (time 1777593188.43 total 30.01 last 119.42) - open/close/unlink 4716 (time 1777593198.44 total 40.01 last 123.88) - open/close/unlink 6152 (time 1777593208.44 total 50.01 last 143.56) - open/close/unlink 7547 (time 1777593218.44 total 60.02 last 139.42) - open/close/unlink 8933 (time 1777593228.45 total 70.02 last 138.54) - open/close/unlink 10000 (time 1777593237.09 total 78.67 last 123.42) - open/close/unlink 11435 (time 1777593247.10 total 88.68 last 143.36) - open/close/unlink 12768 (time 1777593257.11 total 98.68 last 133.28) - open/close/unlink 14083 (time 1777593267.11 total 108.68 last 131.44) - open/close/unlink 15260 (time 1777593277.11 total 118.69 last 117.67) - open/close/unlink 16567 (time 1777593287.11 total 128.69 last 130.68) - open/close/unlink 17878 (time 1777593297.12 total 138.69 last 131.07) - open/close/unlink 19270 (time 1777593307.12 total 148.70 last 139.11) - open/close/unlink 20000 (time 1777593312.18 total 153.75 last 144.38) - open/close/unlink 21508 (time 1777593322.18 total 163.76 last 150.77) - open/close/unlink 23045 (time 1777593332.19 total 173.76 last 153.62) - open/close/unlink 24399 (time 1777593342.19 total 183.76 last 135.39) - open/close/unlink 25763 (time 1777593352.19 total 193.76 last 136.36) - open/close/unlink 27282 (time 1777593362.19 total 203.77 last 151.83) - open/close/unlink 28742 (time 1777593372.19 total 213.77 last 145.99) - open/close/unlink 30000 (time 1777593380.94 total 222.52 last 143.78) - open/close/unlink 31535 (time 1777593390.95 total 232.53 last 153.40) - open/close/unlink 32843 (time 1777593400.96 total 242.53 last 130.70) - open/close/unlink 34242 (time 1777593410.96 total 252.53 last 139.90) - open/close/unlink 35528 (time 1777593420.96 total 262.54 last 128.56) - open/close/unlink 37036 (time 1777593430.97 total 272.54 last 150.69) - open/close/unlink 38573 (time 1777593440.97 total 282.55 last 153.67) - open/close/unlink 40000 (time 1777593450.79 total 292.36 last 145.36) - open/close/unlink 41436 (time 1777593460.79 total 302.37 last 143.56) - open/close/unlink 42925 (time 1777593470.79 total 312.37 last 148.86) - open/close/unlink 44461 (time 1777593480.79 total 322.37 last 153.60) - open/close/unlink 45843 (time 1777593490.80 total 332.37 last 138.12) - open/close/unlink 47443 (time 1777593500.80 total 342.38 last 159.94) - open/close/unlink 49023 (time 1777593510.81 total 352.38 last 157.96) - open/close/unlink 50000 (time 1777593516.54 total 358.11 last 170.53) - open/close/unlink 51639 (time 1777593526.54 total 368.12 last 163.81) - open/close/unlink 53359 (time 1777593536.55 total 378.12 last 171.92) - open/close/unlink 54982 (time 1777593546.55 total 388.12 last 162.26) - open/close/unlink 56507 (time 1777593556.55 total 398.12 last 152.48) - open/close/unlink 58044 (time 1777593566.55 total 408.12 last 153.69) - open/close/unlink 59684 (time 1777593576.55 total 418.12 last 163.99) - open/close/unlink 61393 (time 1777593586.56 total 428.13 last 170.81) - open/close/unlink 62946 (time 1777593596.56 total 438.13 last 155.26) - open/close/unlink 64554 (time 1777593606.57 total 448.14 last 160.68) total: 64768 open/close/unlink in 450.48 seconds: 143.77 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg206-server: cannot open 'lustre-ost2': no such pool pdsh@oleg206-client: oleg206-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg206-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg206-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg206-server stop mds service on oleg206-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg206-server unloading modules via unload_modules_local on: 'oleg206-server' oleg206-server: oleg206-server.virtnet: executing unload_modules_local oleg206-server: modules unloaded. pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-server: ssh exited with exit code 2 pdsh@oleg206-client: oleg206-client: ssh exited with exit code 2