== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 22:41:27 (1786934487) start mds service on oleg254-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg254-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg254-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg254-server: oleg254-server.virtnet: executing set_default_debug -1 all pdsh@oleg254-client: oleg254-server: ssh exited with exit code 1 Started lustre-OST0000 oleg254-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg254-client.virtnet: -o user_xattr,flock 192.168.202.154@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1831 (time 1786934513.33 total 10.00 last 183.03) - open/close/unlink 3522 (time 1786934523.34 total 20.01 last 169.02) - open/close/unlink 5258 (time 1786934533.34 total 30.01 last 173.52) - open/close/unlink 6907 (time 1786934543.35 total 40.02 last 164.82) - open/close/unlink 8602 (time 1786934553.35 total 50.02 last 169.47) - open/close/unlink 10000 (time 1786934561.27 total 57.94 last 176.57) - open/close/unlink 11677 (time 1786934571.27 total 67.94 last 167.69) - open/close/unlink 13353 (time 1786934581.27 total 77.94 last 167.58) - open/close/unlink 15064 (time 1786934591.27 total 87.94 last 171.07) - open/close/unlink 16728 (time 1786934601.27 total 97.94 last 166.36) - open/close/unlink 18443 (time 1786934611.27 total 107.95 last 171.45) - open/close/unlink 20000 (time 1786934621.10 total 117.77 last 158.52) - open/close/unlink 21877 (time 1786934631.10 total 127.77 last 187.68) - open/close/unlink 23685 (time 1786934641.10 total 137.77 last 180.73) - open/close/unlink 25547 (time 1786934651.10 total 147.77 last 186.18) - open/close/unlink 27391 (time 1786934661.10 total 157.77 last 184.39) - open/close/unlink 29221 (time 1786934671.11 total 167.78 last 182.95) - open/close/unlink 30000 (time 1786934675.15 total 171.82 last 192.63) - open/close/unlink 31804 (time 1786934685.16 total 181.83 last 180.29) - open/close/unlink 33641 (time 1786934695.16 total 191.83 last 183.65) - open/close/unlink 35448 (time 1786934705.16 total 201.83 last 180.70) - open/close/unlink 37235 (time 1786934715.16 total 211.84 last 178.61) - open/close/unlink 39016 (time 1786934725.17 total 221.84 last 178.00) - open/close/unlink 40000 (time 1786934730.49 total 227.16 last 184.83) - open/close/unlink 41780 (time 1786934740.49 total 237.17 last 177.97) - open/close/unlink 43603 (time 1786934750.50 total 247.17 last 182.27) - open/close/unlink 45411 (time 1786934760.50 total 257.17 last 180.76) - open/close/unlink 47221 (time 1786934770.50 total 267.17 last 180.96) - open/close/unlink 49088 (time 1786934780.50 total 277.18 last 186.61) - open/close/unlink 50000 (time 1786934785.88 total 282.55 last 169.77) - open/close/unlink 51700 (time 1786934795.88 total 292.55 last 169.95) - open/close/unlink 53530 (time 1786934805.88 total 302.55 last 182.98) - open/close/unlink 55334 (time 1786934815.88 total 312.56 last 180.35) - open/close/unlink 57107 (time 1786934825.88 total 322.56 last 177.29) - open/close/unlink 58919 (time 1786934835.89 total 332.56 last 181.14) - open/close/unlink 60000 (time 1786934841.47 total 338.14 last 193.69) - open/close/unlink 61857 (time 1786934851.47 total 348.15 last 185.59) - open/close/unlink 63713 (time 1786934861.48 total 358.15 last 185.56) total: 64768 open/close/unlink in 363.97 seconds: 177.95 ops/second - open/close/unlink 1835 (time 1786934513.48 total 10.00 last 183.46) - open/close/unlink 3520 (time 1786934523.48 total 20.00 last 168.49) - open/close/unlink 5277 (time 1786934533.48 total 30.00 last 175.66) - open/close/unlink 6938 (time 1786934543.48 total 40.01 last 166.10) - open/close/unlink 8644 (time 1786934553.48 total 50.01 last 170.58) - open/close/unlink 10000 (time 1786934561.18 total 57.70 last 176.25) - open/close/unlink 11693 (time 1786934571.18 total 67.70 last 169.26) - open/close/unlink 13370 (time 1786934581.18 total 77.70 last 167.70) - open/close/unlink 15080 (time 1786934591.18 total 87.70 last 170.98) - open/close/unlink 16754 (time 1786934601.18 total 97.71 last 167.32) - open/close/unlink 18453 (time 1786934611.18 total 107.71 last 169.89) - open/close/unlink 20000 (time 1786934620.99 total 117.52 last 157.74) - open/close/unlink 21873 (time 1786934630.99 total 127.52 last 187.29) - open/close/unlink 23674 (time 1786934641.00 total 137.52 last 180.02) - open/close/unlink 25537 (time 1786934651.00 total 147.52 last 186.29) - open/close/unlink 27380 (time 1786934661.00 total 157.52 last 184.29) - open/close/unlink 29198 (time 1786934671.00 total 167.53 last 181.73) - open/close/unlink 30000 (time 1786934675.19 total 171.71 last 191.49) - open/close/unlink 31797 (time 1786934685.19 total 181.72 last 179.66) - open/close/unlink 33627 (time 1786934695.20 total 191.72 last 182.86) - open/close/unlink 35447 (time 1786934705.21 total 201.73 last 181.90) - open/close/unlink 37231 (time 1786934715.21 total 211.73 last 178.39) - open/close/unlink 39024 (time 1786934725.21 total 221.74 last 179.19) - open/close/unlink 40000 (time 1786934730.52 total 227.05 last 183.87) - open/close/unlink 41793 (time 1786934740.52 total 237.05 last 179.24) - open/close/unlink 43620 (time 1786934750.52 total 247.05 last 182.70) - open/close/unlink 45406 (time 1786934760.53 total 257.05 last 178.56) - open/close/unlink 47215 (time 1786934770.53 total 267.05 last 180.83) - open/close/unlink 49053 (time 1786934780.53 total 277.06 last 183.76) - open/close/unlink 50000 (time 1786934786.08 total 282.60 last 170.84) - open/close/unlink 51714 (time 1786934796.08 total 292.60 last 171.35) - open/close/unlink 53532 (time 1786934806.08 total 302.60 last 181.79) - open/close/unlink 55320 (time 1786934816.08 total 312.61 last 178.75) - open/close/unlink 57090 (time 1786934826.08 total 322.61 last 176.98) - open/close/unlink 58911 (time 1786934836.09 total 332.61 last 182.05) - open/close/unlink 60000 (time 1786934841.70 total 338.22 last 193.99) - open/close/unlink 61869 (time 1786934851.70 total 348.23 last 186.87) - open/close/unlink 63745 (time 1786934861.70 total 358.23 last 187.60) total: 64768 open/close/unlink in 363.84 seconds: 178.01 ops/second - open/close/unlink 1864 (time 1786934513.44 total 10.00 last 186.38) - open/close/unlink 3564 (time 1786934523.44 total 20.00 last 169.98) - open/close/unlink 5332 (time 1786934533.44 total 30.01 last 176.71) - open/close/unlink 6978 (time 1786934543.45 total 40.01 last 164.53) - open/close/unlink 8674 (time 1786934553.45 total 50.01 last 169.60) - open/close/unlink 10000 (time 1786934561.04 total 57.60 last 174.67) - open/close/unlink 11680 (time 1786934571.04 total 67.61 last 167.96) - open/close/unlink 13369 (time 1786934581.04 total 77.61 last 168.86) - open/close/unlink 15085 (time 1786934591.05 total 87.61 last 171.54) - open/close/unlink 16766 (time 1786934601.05 total 97.62 last 168.03) - open/close/unlink 18475 (time 1786934611.05 total 107.62 last 170.86) - open/close/unlink 20000 (time 1786934620.77 total 117.34 last 156.93) - open/close/unlink 21853 (time 1786934630.77 total 127.34 last 185.21) - open/close/unlink 23672 (time 1786934640.78 total 137.34 last 181.86) - open/close/unlink 25517 (time 1786934650.78 total 147.34 last 184.48) - open/close/unlink 27371 (time 1786934660.78 total 157.34 last 185.38) - open/close/unlink 29168 (time 1786934670.79 total 167.35 last 179.55) - open/close/unlink 30000 (time 1786934675.19 total 171.75 last 189.02) - open/close/unlink 31836 (time 1786934685.20 total 181.77 last 183.39) - open/close/unlink 33679 (time 1786934695.20 total 191.77 last 184.22) - open/close/unlink 35490 (time 1786934705.21 total 201.77 last 181.08) - open/close/unlink 37275 (time 1786934715.21 total 211.78 last 178.43) - open/close/unlink 39058 (time 1786934725.21 total 221.78 last 178.29) - open/close/unlink 40000 (time 1786934730.42 total 226.98 last 180.96) - open/close/unlink 41796 (time 1786934740.42 total 236.99 last 179.53) - open/close/unlink 43609 (time 1786934750.42 total 246.99 last 181.29) - open/close/unlink 45394 (time 1786934760.42 total 256.99 last 178.47) - open/close/unlink 47202 (time 1786934770.42 total 266.99 last 180.78) - open/close/unlink 49052 (time 1786934780.43 total 276.99 last 184.93) - open/close/unlink 50000 (time 1786934785.91 total 282.48 last 172.87) - open/close/unlink 51707 (time 1786934795.91 total 292.48 last 170.67) - open/close/unlink 53528 (time 1786934805.91 total 302.48 last 182.08) - open/close/unlink 55316 (time 1786934815.91 total 312.48 last 178.79) - open/close/unlink 57067 (time 1786934825.92 total 322.48 last 175.01) - open/close/unlink 58882 (time 1786934835.92 total 332.49 last 181.46) - open/close/unlink 60000 (time 1786934841.72 total 338.28 last 192.95) - open/close/unlink 61872 (time 1786934851.72 total 348.29 last 187.13) - open/close/unlink 63737 (time 1786934861.72 total 358.29 last 186.45) total: 64768 open/close/unlink in 364.15 seconds: 177.86 ops/second - open/close/unlink 1835 (time 1786934513.43 total 10.00 last 183.48) - open/close/unlink 3513 (time 1786934523.43 total 20.00 last 167.80) - open/close/unlink 5260 (time 1786934533.44 total 30.01 last 174.61) - open/close/unlink 6907 (time 1786934543.44 total 40.01 last 164.64) - open/close/unlink 8613 (time 1786934553.45 total 50.02 last 170.52) - open/close/unlink 10000 (time 1786934561.28 total 57.85 last 176.99) - open/close/unlink 11686 (time 1786934571.28 total 67.85 last 168.59) - open/close/unlink 13362 (time 1786934581.28 total 77.85 last 167.58) - open/close/unlink 15073 (time 1786934591.29 total 87.86 last 171.06) - open/close/unlink 16734 (time 1786934601.29 total 97.86 last 166.08) - open/close/unlink 18453 (time 1786934611.29 total 107.86 last 171.89) - open/close/unlink 20000 (time 1786934621.04 total 117.61 last 158.56) - open/close/unlink 21853 (time 1786934631.05 total 127.62 last 185.25) - open/close/unlink 23661 (time 1786934641.05 total 137.62 last 180.76) - open/close/unlink 25512 (time 1786934651.05 total 147.62 last 185.09) - open/close/unlink 27355 (time 1786934661.05 total 157.62 last 184.28) - open/close/unlink 29160 (time 1786934671.05 total 167.62 last 180.49) - open/close/unlink 30000 (time 1786934675.68 total 172.25 last 181.39) - open/close/unlink 31809 (time 1786934685.69 total 182.26 last 180.82) - open/close/unlink 33673 (time 1786934695.69 total 192.26 last 186.33) - open/close/unlink 35491 (time 1786934705.70 total 202.27 last 181.69) - open/close/unlink 37291 (time 1786934715.71 total 212.28 last 179.73) - open/close/unlink 39103 (time 1786934725.72 total 222.29 last 181.12) - open/close/unlink 40000 (time 1786934730.67 total 227.24 last 181.14) - open/close/unlink 41797 (time 1786934740.67 total 237.24 last 179.67) - open/close/unlink 43621 (time 1786934750.67 total 247.24 last 182.34) - open/close/unlink 45421 (time 1786934760.67 total 257.24 last 179.99) - open/close/unlink 47240 (time 1786934770.67 total 267.24 last 181.89) - open/close/unlink 49078 (time 1786934780.68 total 277.25 last 183.74) - open/close/unlink 50000 (time 1786934786.10 total 282.67 last 170.06) - open/close/unlink 51710 (time 1786934796.10 total 292.67 last 170.95) - open/close/unlink 53542 (time 1786934806.11 total 302.68 last 183.12) - open/close/unlink 55328 (time 1786934816.11 total 312.68 last 178.56) - open/close/unlink 57077 (time 1786934826.11 total 322.68 last 174.90) - open/close/unlink 58890 (time 1786934836.11 total 332.68 last 181.27) - open/close/unlink 60000 (time 1786934841.82 total 338.39 last 194.39) - open/close/unlink 61869 (time 1786934851.83 total 348.40 last 186.82) - open/close/unlink 63725 (time 1786934861.83 total 358.40 last 185.58) total: 64768 open/close/unlink in 364.43 seconds: 177.72 ops/second - open/close/unlink 1848 (time 1786934513.48 total 10.00 last 184.76) - open/close/unlink 3532 (time 1786934523.48 total 20.00 last 168.40) - open/close/unlink 5295 (time 1786934533.48 total 30.00 last 176.28) - open/close/unlink 6955 (time 1786934543.49 total 40.00 last 165.98) - open/close/unlink 8656 (time 1786934553.49 total 50.01 last 170.06) - open/close/unlink 10000 (time 1786934561.14 total 57.66 last 175.57) - open/close/unlink 11665 (time 1786934571.14 total 67.66 last 166.48) - open/close/unlink 13338 (time 1786934581.15 total 77.67 last 167.25) - open/close/unlink 15042 (time 1786934591.15 total 87.67 last 170.36) - open/close/unlink 16729 (time 1786934601.15 total 97.67 last 168.62) - open/close/unlink 18417 (time 1786934611.15 total 107.67 last 168.79) - open/close/unlink 20000 (time 1786934621.08 total 117.60 last 159.55) - open/close/unlink 21874 (time 1786934631.08 total 127.60 last 187.38) - open/close/unlink 23676 (time 1786934641.08 total 137.60 last 180.14) - open/close/unlink 25536 (time 1786934651.08 total 147.60 last 185.96) - open/close/unlink 27386 (time 1786934661.09 total 157.60 last 184.95) - open/close/unlink 29193 (time 1786934671.09 total 167.61 last 180.66) - open/close/unlink 30000 (time 1786934675.52 total 172.04 last 181.87) - open/close/unlink 31802 (time 1786934685.53 total 182.05 last 180.04) - open/close/unlink 33637 (time 1786934695.54 total 192.06 last 183.41) - open/close/unlink 35458 (time 1786934705.54 total 202.06 last 182.04) - open/close/unlink 37262 (time 1786934715.55 total 212.07 last 180.30) - open/close/unlink 39061 (time 1786934725.55 total 222.07 last 179.88) - open/close/unlink 40000 (time 1786934730.69 total 227.21 last 182.48) - open/close/unlink 41789 (time 1786934740.69 total 237.21 last 178.90) - open/close/unlink 43603 (time 1786934750.70 total 247.22 last 181.34) - open/close/unlink 45398 (time 1786934760.70 total 257.22 last 179.49) - open/close/unlink 47225 (time 1786934770.70 total 267.22 last 182.66) - open/close/unlink 49066 (time 1786934780.70 total 277.22 last 184.08) - open/close/unlink 50000 (time 1786934786.12 total 282.64 last 172.22) - open/close/unlink 51696 (time 1786934796.13 total 292.65 last 169.57) - open/close/unlink 53535 (time 1786934806.13 total 302.65 last 183.83) - open/close/unlink 55326 (time 1786934816.13 total 312.65 last 179.05) - open/close/unlink 57071 (time 1786934826.14 total 322.66 last 174.44) - open/close/unlink 58883 (time 1786934836.14 total 332.66 last 181.13) - open/close/unlink 60000 (time 1786934841.91 total 338.43 last 193.47) - open/close/unlink 61880 (time 1786934851.91 total 348.43 last 187.98) - open/close/unlink 63731 (time 1786934861.92 total 358.44 last 185.05) total: 64768 open/close/unlink in 364.44 seconds: 177.72 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg254-server: cannot open 'lustre-ost2': no such pool pdsh@oleg254-client: oleg254-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg254-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg254-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg254-server stop mds service on oleg254-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg254-server unloading modules via unload_modules_local on: 'oleg254-server' oleg254-server: oleg254-server.virtnet: executing unload_modules_local oleg254-server: modules unloaded. pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-client: ssh exited with exit code 2 pdsh@oleg254-client: oleg254-server: ssh exited with exit code 2