== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 04:51:35 (1785315095) start mds service on oleg635-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg635-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg635-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg635-server: oleg635-server.virtnet: executing set_default_debug -1 all pdsh@oleg635-client: oleg635-server: ssh exited with exit code 1 Started lustre-OST0000 oleg635-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg635-client.virtnet: -o user_xattr,flock 192.168.206.135@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92577 330 92247 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12234 330 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1690 (time 1785315121.39 total 10.00 last 168.93) - open/close/unlink 3441 (time 1785315131.39 total 20.01 last 175.06) - open/close/unlink 5148 (time 1785315141.39 total 30.01 last 170.63) - open/close/unlink 6577 (time 1785315151.40 total 40.01 last 142.84) - open/close/unlink 7937 (time 1785315161.40 total 50.02 last 135.98) - open/close/unlink 9307 (time 1785315171.40 total 60.02 last 136.93) - open/close/unlink 10000 (time 1785315176.55 total 65.17 last 134.67) - open/close/unlink 11351 (time 1785315186.55 total 75.17 last 135.08) - open/close/unlink 13113 (time 1785315196.55 total 85.17 last 176.16) - open/close/unlink 14869 (time 1785315206.56 total 95.17 last 175.54) - open/close/unlink 16501 (time 1785315216.56 total 105.18 last 163.17) - open/close/unlink 17783 (time 1785315226.56 total 115.18 last 128.20) - open/close/unlink 19295 (time 1785315236.57 total 125.18 last 151.08) - open/close/unlink 20000 (time 1785315240.60 total 129.22 last 174.73) - open/close/unlink 21594 (time 1785315250.60 total 139.22 last 159.38) - open/close/unlink 23241 (time 1785315260.60 total 149.22 last 164.68) - open/close/unlink 24659 (time 1785315270.61 total 159.23 last 141.72) - open/close/unlink 26193 (time 1785315280.61 total 169.23 last 153.34) - open/close/unlink 27939 (time 1785315290.61 total 179.23 last 174.59) - open/close/unlink 29607 (time 1785315300.62 total 189.24 last 166.72) - open/close/unlink 30000 (time 1785315302.74 total 191.36 last 185.17) - open/close/unlink 31735 (time 1785315312.74 total 201.36 last 173.44) - open/close/unlink 33494 (time 1785315322.75 total 211.37 last 175.82) - open/close/unlink 35208 (time 1785315332.75 total 221.37 last 171.40) - open/close/unlink 36632 (time 1785315342.75 total 231.37 last 142.38) - open/close/unlink 38313 (time 1785315352.75 total 241.37 last 168.06) - open/close/unlink 39992 (time 1785315362.76 total 251.37 last 167.84) - open/close/unlink 41805 (time 1785315372.76 total 261.38 last 181.25) - open/close/unlink 43461 (time 1785315382.76 total 271.38 last 165.59) - open/close/unlink 44912 (time 1785315392.76 total 281.38 last 145.08) - open/close/unlink 46394 (time 1785315402.76 total 291.38 last 148.19) - open/close/unlink 48125 (time 1785315412.76 total 301.38 last 173.08) - open/close/unlink 49752 (time 1785315422.77 total 311.38 last 162.65) - open/close/unlink 51435 (time 1785315432.77 total 321.39 last 168.26) - open/close/unlink 52787 (time 1785315442.77 total 331.39 last 135.20) - open/close/unlink 54057 (time 1785315452.77 total 341.39 last 126.93) - open/close/unlink 55455 (time 1785315462.77 total 351.39 last 139.80) - open/close/unlink 56781 (time 1785315472.78 total 361.39 last 132.59) - open/close/unlink 58138 (time 1785315482.78 total 371.40 last 135.61) - open/close/unlink 59687 (time 1785315492.79 total 381.40 last 154.81) - open/close/unlink 61389 (time 1785315502.79 total 391.40 last 170.20) - open/close/unlink 63178 (time 1785315512.79 total 401.41 last 178.86) total: 64768 open/close/unlink in 410.43 seconds: 157.81 ops/second - open/close/unlink 1664 (time 1785315121.06 total 10.00 last 166.35) - open/close/unlink 3413 (time 1785315131.06 total 20.01 last 174.83) - open/close/unlink 5125 (time 1785315141.07 total 30.01 last 171.08) - open/close/unlink 6540 (time 1785315151.08 total 40.02 last 141.44) - open/close/unlink 7902 (time 1785315161.08 total 50.02 last 136.16) - open/close/unlink 9274 (time 1785315171.08 total 60.03 last 137.13) - open/close/unlink 10000 (time 1785315176.41 total 65.35 last 136.31) - open/close/unlink 11349 (time 1785315186.41 total 75.36 last 134.86) - open/close/unlink 13119 (time 1785315196.41 total 85.36 last 176.96) - open/close/unlink 14886 (time 1785315206.42 total 95.36 last 176.62) - open/close/unlink 16524 (time 1785315216.44 total 105.39 last 163.41) - open/close/unlink 17804 (time 1785315226.45 total 115.40 last 127.86) - open/close/unlink 19317 (time 1785315236.46 total 125.41 last 151.17) - open/close/unlink 20000 (time 1785315240.42 total 129.36 last 172.60) - open/close/unlink 21578 (time 1785315250.42 total 139.36 last 157.79) - open/close/unlink 23232 (time 1785315260.42 total 149.37 last 165.35) - open/close/unlink 24645 (time 1785315270.43 total 159.37 last 141.22) - open/close/unlink 26178 (time 1785315280.43 total 169.37 last 153.26) - open/close/unlink 27921 (time 1785315290.43 total 179.38 last 174.24) - open/close/unlink 29595 (time 1785315300.44 total 189.38 last 167.38) - open/close/unlink 30000 (time 1785315302.50 total 191.45 last 195.84) - open/close/unlink 31745 (time 1785315312.51 total 201.45 last 174.44) - open/close/unlink 33476 (time 1785315322.51 total 211.46 last 173.01) - open/close/unlink 35201 (time 1785315332.52 total 221.46 last 172.40) - open/close/unlink 36624 (time 1785315342.52 total 231.47 last 142.23) - open/close/unlink 38305 (time 1785315352.53 total 241.47 last 168.04) - open/close/unlink 39983 (time 1785315362.53 total 251.47 last 167.76) - open/close/unlink 41795 (time 1785315372.53 total 261.48 last 181.13) - open/close/unlink 43432 (time 1785315382.54 total 271.48 last 163.63) - open/close/unlink 44893 (time 1785315392.54 total 281.48 last 146.06) - open/close/unlink 46361 (time 1785315402.54 total 291.48 last 146.79) - open/close/unlink 48090 (time 1785315412.54 total 301.49 last 172.85) - open/close/unlink 49740 (time 1785315422.55 total 311.49 last 164.93) - open/close/unlink 51415 (time 1785315432.55 total 321.49 last 167.46) - open/close/unlink 52754 (time 1785315442.56 total 331.51 last 133.70) - open/close/unlink 53999 (time 1785315452.57 total 341.51 last 124.49) - open/close/unlink 55401 (time 1785315462.57 total 351.51 last 140.19) - open/close/unlink 56742 (time 1785315472.57 total 361.51 last 134.04) - open/close/unlink 58089 (time 1785315482.58 total 371.52 last 134.61) - open/close/unlink 59651 (time 1785315492.58 total 381.52 last 156.16) - open/close/unlink 60000 (time 1785315494.68 total 383.62 last 166.41) - open/close/unlink 61684 (time 1785315504.68 total 393.62 last 168.35) - open/close/unlink 63455 (time 1785315514.68 total 403.62 last 177.08) total: 64768 open/close/unlink in 411.37 seconds: 157.44 ops/second - open/close/unlink 1677 (time 1785315121.29 total 10.00 last 167.63) - open/close/unlink 3418 (time 1785315131.30 total 20.01 last 174.04) - open/close/unlink 5139 (time 1785315141.30 total 30.01 last 172.05) - open/close/unlink 6553 (time 1785315151.30 total 40.01 last 141.34) - open/close/unlink 7907 (time 1785315161.31 total 50.02 last 135.38) - open/close/unlink 9275 (time 1785315171.31 total 60.02 last 136.78) - open/close/unlink 10000 (time 1785315176.55 total 65.26 last 138.37) - open/close/unlink 11347 (time 1785315186.55 total 75.26 last 134.69) - open/close/unlink 13115 (time 1785315196.55 total 85.26 last 176.74) - open/close/unlink 14857 (time 1785315206.55 total 95.26 last 174.14) - open/close/unlink 16479 (time 1785315216.56 total 105.28 last 162.03) - open/close/unlink 17763 (time 1785315226.57 total 115.28 last 128.39) - open/close/unlink 19292 (time 1785315236.58 total 125.29 last 152.75) - open/close/unlink 20000 (time 1785315240.63 total 129.34 last 174.71) - open/close/unlink 21592 (time 1785315250.63 total 139.34 last 159.11) - open/close/unlink 23242 (time 1785315260.64 total 149.35 last 164.95) - open/close/unlink 24651 (time 1785315270.64 total 159.35 last 140.86) - open/close/unlink 26194 (time 1785315280.64 total 169.35 last 154.24) - open/close/unlink 27930 (time 1785315290.65 total 179.36 last 173.55) - open/close/unlink 29597 (time 1785315300.65 total 189.36 last 166.65) - open/close/unlink 30000 (time 1785315302.88 total 191.59 last 180.32) - open/close/unlink 31730 (time 1785315312.90 total 201.61 last 172.78) - open/close/unlink 33527 (time 1785315322.90 total 211.61 last 179.67) - open/close/unlink 35234 (time 1785315332.90 total 221.61 last 170.65) - open/close/unlink 36655 (time 1785315342.91 total 231.62 last 142.05) - open/close/unlink 38339 (time 1785315352.91 total 241.62 last 168.39) - open/close/unlink 40000 (time 1785315362.81 total 251.52 last 167.71) - open/close/unlink 41815 (time 1785315372.81 total 261.52 last 181.44) - open/close/unlink 43449 (time 1785315382.81 total 271.52 last 163.40) - open/close/unlink 44905 (time 1785315392.82 total 281.53 last 145.57) - open/close/unlink 46383 (time 1785315402.82 total 291.53 last 147.79) - open/close/unlink 48115 (time 1785315412.82 total 301.53 last 173.18) - open/close/unlink 49758 (time 1785315422.82 total 311.53 last 164.25) - open/close/unlink 51434 (time 1785315432.82 total 321.53 last 167.55) - open/close/unlink 52770 (time 1785315442.83 total 331.54 last 133.49) - open/close/unlink 54029 (time 1785315452.84 total 341.55 last 125.83) - open/close/unlink 55430 (time 1785315462.84 total 351.55 last 140.06) - open/close/unlink 56752 (time 1785315472.84 total 361.55 last 132.19) - open/close/unlink 58104 (time 1785315482.84 total 371.55 last 135.19) - open/close/unlink 59651 (time 1785315492.85 total 381.56 last 154.59) - open/close/unlink 60000 (time 1785315494.88 total 383.60 last 171.41) - open/close/unlink 61670 (time 1785315504.89 total 393.60 last 166.92) - open/close/unlink 63461 (time 1785315514.89 total 403.60 last 179.07) total: 64768 open/close/unlink in 413.30 seconds: 156.71 ops/second - open/close/unlink 1694 (time 1785315121.42 total 10.00 last 169.34) - open/close/unlink 3426 (time 1785315131.42 total 20.01 last 173.17) - open/close/unlink 5124 (time 1785315141.42 total 30.01 last 169.78) - open/close/unlink 6543 (time 1785315151.43 total 40.01 last 141.85) - open/close/unlink 7914 (time 1785315161.43 total 50.01 last 137.04) - open/close/unlink 9277 (time 1785315171.43 total 60.02 last 136.28) - open/close/unlink 10000 (time 1785315176.76 total 65.34 last 135.79) - open/close/unlink 11350 (time 1785315186.76 total 75.34 last 135.00) - open/close/unlink 13135 (time 1785315196.76 total 85.34 last 178.45) - open/close/unlink 14876 (time 1785315206.76 total 95.35 last 174.06) - open/close/unlink 16464 (time 1785315216.77 total 105.35 last 158.73) - open/close/unlink 17745 (time 1785315226.77 total 115.35 last 128.04) - open/close/unlink 19259 (time 1785315236.78 total 125.36 last 151.36) - open/close/unlink 20000 (time 1785315240.92 total 129.50 last 178.97) - open/close/unlink 21616 (time 1785315250.92 total 139.50 last 161.52) - open/close/unlink 23246 (time 1785315260.92 total 149.51 last 162.93) - open/close/unlink 24645 (time 1785315270.93 total 159.51 last 139.81) - open/close/unlink 26203 (time 1785315280.93 total 169.52 last 155.75) - open/close/unlink 27954 (time 1785315290.94 total 179.52 last 175.08) - open/close/unlink 29599 (time 1785315300.94 total 189.52 last 164.46) - open/close/unlink 30000 (time 1785315303.43 total 192.02 last 160.64) - open/close/unlink 31726 (time 1785315313.44 total 202.02 last 172.51) - open/close/unlink 33501 (time 1785315323.45 total 212.03 last 177.39) - open/close/unlink 35219 (time 1785315333.46 total 222.04 last 171.62) - open/close/unlink 36659 (time 1785315343.46 total 232.04 last 143.94) - open/close/unlink 38382 (time 1785315353.46 total 242.05 last 172.23) - open/close/unlink 40000 (time 1785315363.14 total 251.72 last 167.22) - open/close/unlink 41801 (time 1785315373.14 total 261.72 last 180.05) - open/close/unlink 43429 (time 1785315383.15 total 271.73 last 162.75) - open/close/unlink 44864 (time 1785315393.15 total 281.73 last 143.44) - open/close/unlink 46361 (time 1785315403.15 total 291.73 last 149.69) - open/close/unlink 48073 (time 1785315413.15 total 301.73 last 171.17) - open/close/unlink 49711 (time 1785315423.16 total 311.74 last 163.75) - open/close/unlink 51371 (time 1785315433.16 total 321.74 last 165.91) - open/close/unlink 52670 (time 1785315443.16 total 331.75 last 129.86) - open/close/unlink 53956 (time 1785315453.17 total 341.75 last 128.55) - open/close/unlink 55367 (time 1785315463.17 total 351.75 last 141.08) - open/close/unlink 56689 (time 1785315473.17 total 361.76 last 132.13) - open/close/unlink 58028 (time 1785315483.18 total 371.76 last 133.85) - open/close/unlink 59602 (time 1785315493.18 total 381.76 last 157.37) - open/close/unlink 60000 (time 1785315495.45 total 384.03 last 175.59) - open/close/unlink 61701 (time 1785315505.45 total 394.03 last 170.09) - open/close/unlink 63491 (time 1785315515.45 total 404.03 last 178.93) - open/close/unlink 64717 (time 1785315525.46 total 414.04 last 122.51) total: 64768 open/close/unlink in 415.66 seconds: 155.82 ops/second - open/close/unlink 1697 (time 1785315121.44 total 10.01 last 169.60) - open/close/unlink 3434 (time 1785315131.44 total 20.01 last 173.64) - open/close/unlink 5145 (time 1785315141.44 total 30.01 last 171.06) - open/close/unlink 6560 (time 1785315151.44 total 40.01 last 141.48) - open/close/unlink 7917 (time 1785315161.45 total 50.02 last 135.62) - open/close/unlink 9281 (time 1785315171.45 total 60.02 last 136.38) - open/close/unlink 10000 (time 1785315176.74 total 65.31 last 135.89) - open/close/unlink 11351 (time 1785315186.75 total 75.32 last 135.04) - open/close/unlink 13111 (time 1785315196.75 total 85.32 last 175.92) - open/close/unlink 14861 (time 1785315206.75 total 95.32 last 174.98) - open/close/unlink 16460 (time 1785315216.76 total 105.32 last 159.86) - open/close/unlink 17770 (time 1785315226.76 total 115.33 last 130.97) - open/close/unlink 19290 (time 1785315236.76 total 125.33 last 151.93) - open/close/unlink 20000 (time 1785315240.79 total 129.36 last 176.32) - open/close/unlink 21602 (time 1785315250.79 total 139.36 last 160.20) - open/close/unlink 23228 (time 1785315260.79 total 149.36 last 162.57) - open/close/unlink 24637 (time 1785315270.80 total 159.36 last 140.84) - open/close/unlink 26177 (time 1785315280.80 total 169.37 last 153.91) - open/close/unlink 27902 (time 1785315290.80 total 179.37 last 172.49) - open/close/unlink 29563 (time 1785315300.80 total 189.37 last 166.07) - open/close/unlink 30000 (time 1785315303.50 total 192.07 last 162.21) - open/close/unlink 31726 (time 1785315313.50 total 202.07 last 172.54) - open/close/unlink 33508 (time 1785315323.50 total 212.07 last 178.17) - open/close/unlink 35229 (time 1785315333.51 total 222.07 last 172.06) - open/close/unlink 36661 (time 1785315343.51 total 232.08 last 143.14) - open/close/unlink 38355 (time 1785315353.52 total 242.08 last 169.29) - open/close/unlink 40000 (time 1785315363.23 total 251.80 last 169.26) - open/close/unlink 41804 (time 1785315373.24 total 261.80 last 180.35) - open/close/unlink 43431 (time 1785315383.24 total 271.81 last 162.64) - open/close/unlink 44871 (time 1785315393.24 total 281.81 last 143.95) - open/close/unlink 46381 (time 1785315403.25 total 291.82 last 150.94) - open/close/unlink 48099 (time 1785315413.25 total 301.82 last 171.73) - open/close/unlink 49752 (time 1785315423.25 total 311.82 last 165.26) - open/close/unlink 51413 (time 1785315433.26 total 321.83 last 165.98) - open/close/unlink 52711 (time 1785315443.27 total 331.84 last 129.70) - open/close/unlink 53993 (time 1785315453.27 total 341.84 last 128.20) - open/close/unlink 55409 (time 1785315463.27 total 351.84 last 141.54) - open/close/unlink 56730 (time 1785315473.28 total 361.84 last 132.08) - open/close/unlink 58054 (time 1785315483.28 total 371.84 last 132.38) - open/close/unlink 59630 (time 1785315493.28 total 381.85 last 157.54) - open/close/unlink 60000 (time 1785315495.42 total 383.99 last 172.74) - open/close/unlink 61692 (time 1785315505.43 total 394.00 last 169.12) - open/close/unlink 63470 (time 1785315515.43 total 404.00 last 177.76) - open/close/unlink 64695 (time 1785315525.46 total 414.03 last 122.11) total: 64768 open/close/unlink in 416.23 seconds: 155.61 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg635-server: cannot open 'lustre-ost2': no such pool pdsh@oleg635-client: oleg635-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg635-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg635-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg635-server stop mds service on oleg635-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg635-server unloading modules via unload_modules_local on: 'oleg635-server' oleg635-server: oleg635-server.virtnet: executing unload_modules_local oleg635-server: modules unloaded. pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-server: ssh exited with exit code 2 pdsh@oleg635-client: oleg635-client: ssh exited with exit code 2