== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 19:39:53 (1788737993) start mds service on oleg357-server Start mds1: mount -t lustre -o localrecov lustre-mdt1/mdt1 /mnt/lustre-mds1 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-MDT0000 oleg357-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg357-server Start ost1: mount -t lustre -o localrecov lustre-ost1/ost1 /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg357-server: oleg357-server.virtnet: executing set_default_debug -1 all pdsh@oleg357-client: oleg357-server: ssh exited with exit code 1 Started lustre-OST0000 oleg357-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg357-client.virtnet: -o user_xattr,flock 192.168.203.157@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 92553 329 92224 1% /mnt/lustre[MDT:0] lustre-OST0000_UUID 12329 425 11904 4% /mnt/lustre[OST:0] filesystem_summary: 12233 329 11904 3% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1697 (time 1788738024.90 total 10.00 last 169.63) - open/close/unlink 3276 (time 1788738034.90 total 20.01 last 157.83) - open/close/unlink 4961 (time 1788738044.90 total 30.01 last 168.46) - open/close/unlink 6762 (time 1788738054.91 total 40.02 last 180.00) - open/close/unlink 8578 (time 1788738064.91 total 50.02 last 181.53) - open/close/unlink 10000 (time 1788738072.59 total 57.70 last 185.22) - open/close/unlink 11720 (time 1788738082.59 total 67.70 last 171.98) - open/close/unlink 13515 (time 1788738092.62 total 77.73 last 179.01) - open/close/unlink 15267 (time 1788738102.62 total 87.73 last 175.18) - open/close/unlink 16964 (time 1788738112.63 total 97.73 last 169.64) - open/close/unlink 18725 (time 1788738122.63 total 107.73 last 176.08) - open/close/unlink 20000 (time 1788738129.93 total 115.03 last 174.60) - open/close/unlink 21816 (time 1788738139.93 total 125.04 last 181.54) - open/close/unlink 23644 (time 1788738149.94 total 135.04 last 182.70) - open/close/unlink 25573 (time 1788738159.94 total 145.04 last 192.88) - open/close/unlink 27490 (time 1788738169.94 total 155.05 last 191.63) - open/close/unlink 29378 (time 1788738179.95 total 165.05 last 188.67) - open/close/unlink 30000 (time 1788738183.44 total 168.54 last 178.36) - open/close/unlink 31857 (time 1788738193.44 total 178.55 last 185.64) - open/close/unlink 33674 (time 1788738203.44 total 188.55 last 181.63) - open/close/unlink 35551 (time 1788738213.44 total 198.55 last 187.68) - open/close/unlink 37431 (time 1788738223.45 total 208.55 last 187.95) - open/close/unlink 39336 (time 1788738233.45 total 218.55 last 190.49) - open/close/unlink 40000 (time 1788738237.05 total 222.15 last 184.46) - open/close/unlink 41773 (time 1788738247.06 total 232.17 last 177.04) - open/close/unlink 43530 (time 1788738257.07 total 242.17 last 175.63) - open/close/unlink 45261 (time 1788738267.07 total 252.17 last 173.10) - open/close/unlink 47190 (time 1788738277.07 total 262.17 last 192.88) - open/close/unlink 49117 (time 1788738287.07 total 272.18 last 192.59) - open/close/unlink 50000 (time 1788738293.19 total 278.30 last 144.33) - open/close/unlink 51908 (time 1788738303.20 total 288.30 last 190.64) - open/close/unlink 53800 (time 1788738313.20 total 298.31 last 189.13) - open/close/unlink 55662 (time 1788738323.21 total 308.31 last 186.11) - open/close/unlink 57491 (time 1788738333.21 total 318.32 last 182.82) - open/close/unlink 59352 (time 1788738343.21 total 328.32 last 186.10) - open/close/unlink 60000 (time 1788738346.81 total 331.91 last 180.18) - open/close/unlink 61880 (time 1788738356.81 total 341.92 last 187.92) - open/close/unlink 63732 (time 1788738366.81 total 351.92 last 185.17) total: 64768 open/close/unlink in 357.37 seconds: 181.24 ops/second - open/close/unlink 1697 (time 1788738024.81 total 10.00 last 169.68) - open/close/unlink 3277 (time 1788738034.81 total 20.01 last 157.93) - open/close/unlink 4985 (time 1788738044.81 total 30.01 last 170.74) - open/close/unlink 6777 (time 1788738054.82 total 40.01 last 179.16) - open/close/unlink 8594 (time 1788738064.82 total 50.01 last 181.68) - open/close/unlink 10000 (time 1788738072.40 total 57.60 last 185.33) - open/close/unlink 11711 (time 1788738082.41 total 67.61 last 170.99) - open/close/unlink 13505 (time 1788738092.42 total 77.61 last 179.29) - open/close/unlink 15229 (time 1788738102.42 total 87.62 last 172.31) - open/close/unlink 16931 (time 1788738112.42 total 97.62 last 170.17) - open/close/unlink 18716 (time 1788738122.43 total 107.62 last 178.45) - open/close/unlink 20000 (time 1788738129.83 total 115.02 last 173.52) - open/close/unlink 21819 (time 1788738139.84 total 125.04 last 181.64) - open/close/unlink 23631 (time 1788738149.85 total 135.04 last 181.05) - open/close/unlink 25537 (time 1788738159.86 total 145.05 last 190.40) - open/close/unlink 27454 (time 1788738169.86 total 155.06 last 191.68) - open/close/unlink 29357 (time 1788738179.87 total 165.06 last 190.20) - open/close/unlink 30000 (time 1788738183.49 total 168.68 last 177.46) - open/close/unlink 31856 (time 1788738193.49 total 178.69 last 185.57) - open/close/unlink 33673 (time 1788738203.49 total 188.69 last 181.65) - open/close/unlink 35562 (time 1788738213.49 total 198.69 last 188.88) - open/close/unlink 37451 (time 1788738223.50 total 208.69 last 188.84) - open/close/unlink 39342 (time 1788738233.50 total 218.69 last 189.07) - open/close/unlink 40000 (time 1788738236.94 total 222.13 last 191.27) - open/close/unlink 41771 (time 1788738246.94 total 232.14 last 177.01) - open/close/unlink 43528 (time 1788738256.95 total 242.15 last 175.58) - open/close/unlink 45262 (time 1788738266.95 total 252.15 last 173.38) - open/close/unlink 47175 (time 1788738276.95 total 262.15 last 191.29) - open/close/unlink 49088 (time 1788738286.95 total 272.15 last 191.29) - open/close/unlink 50000 (time 1788738293.26 total 278.45 last 144.67) - open/close/unlink 51911 (time 1788738303.26 total 288.46 last 191.01) - open/close/unlink 53804 (time 1788738313.27 total 298.46 last 189.22) - open/close/unlink 55662 (time 1788738323.27 total 308.47 last 185.70) - open/close/unlink 57505 (time 1788738333.27 total 318.47 last 184.26) - open/close/unlink 59363 (time 1788738343.27 total 328.47 last 185.77) - open/close/unlink 60000 (time 1788738346.80 total 332.00 last 180.53) - open/close/unlink 61877 (time 1788738356.81 total 342.00 last 187.58) - open/close/unlink 63739 (time 1788738366.81 total 352.01 last 186.13) total: 64768 open/close/unlink in 357.47 seconds: 181.19 ops/second - open/close/unlink 1667 (time 1788738024.45 total 10.00 last 166.65) - open/close/unlink 3254 (time 1788738034.45 total 20.00 last 158.70) - open/close/unlink 4955 (time 1788738044.46 total 30.01 last 170.02) - open/close/unlink 6737 (time 1788738054.46 total 40.01 last 178.20) - open/close/unlink 8553 (time 1788738064.46 total 50.01 last 181.59) - open/close/unlink 10000 (time 1788738072.14 total 57.69 last 188.30) - open/close/unlink 11702 (time 1788738082.15 total 67.70 last 170.14) - open/close/unlink 13484 (time 1788738092.15 total 77.70 last 178.17) - open/close/unlink 15200 (time 1788738102.16 total 87.71 last 171.46) - open/close/unlink 16898 (time 1788738112.16 total 97.71 last 169.77) - open/close/unlink 18658 (time 1788738122.16 total 107.71 last 175.95) - open/close/unlink 20000 (time 1788738129.77 total 115.32 last 176.41) - open/close/unlink 21844 (time 1788738139.77 total 125.32 last 184.36) - open/close/unlink 23665 (time 1788738149.77 total 135.33 last 182.00) - open/close/unlink 25585 (time 1788738159.78 total 145.33 last 191.99) - open/close/unlink 27507 (time 1788738169.78 total 155.33 last 192.10) - open/close/unlink 29414 (time 1788738179.78 total 165.33 last 190.65) - open/close/unlink 30000 (time 1788738183.22 total 168.77 last 170.37) - open/close/unlink 31840 (time 1788738193.23 total 178.78 last 183.93) - open/close/unlink 33669 (time 1788738203.23 total 188.78 last 182.88) - open/close/unlink 35536 (time 1788738213.23 total 198.78 last 186.67) - open/close/unlink 37421 (time 1788738223.23 total 208.78 last 188.43) - open/close/unlink 39314 (time 1788738233.23 total 218.78 last 189.30) - open/close/unlink 40000 (time 1788738236.82 total 222.37 last 191.16) - open/close/unlink 41768 (time 1788738246.83 total 232.38 last 176.68) - open/close/unlink 43512 (time 1788738256.83 total 242.38 last 174.32) - open/close/unlink 45215 (time 1788738266.83 total 252.38 last 170.30) - open/close/unlink 47126 (time 1788738276.84 total 262.39 last 191.03) - open/close/unlink 49032 (time 1788738286.84 total 272.39 last 190.58) - open/close/unlink 50000 (time 1788738293.39 total 278.94 last 147.73) - open/close/unlink 51894 (time 1788738303.40 total 288.95 last 189.27) - open/close/unlink 53773 (time 1788738313.40 total 298.95 last 187.85) - open/close/unlink 55641 (time 1788738323.41 total 308.96 last 186.60) - open/close/unlink 57491 (time 1788738333.41 total 318.96 last 184.95) - open/close/unlink 59356 (time 1788738343.42 total 328.97 last 186.34) - open/close/unlink 60000 (time 1788738347.07 total 332.62 last 176.75) - open/close/unlink 61885 (time 1788738357.07 total 342.62 last 188.50) - open/close/unlink 63742 (time 1788738367.07 total 352.62 last 185.69) total: 64768 open/close/unlink in 358.24 seconds: 180.79 ops/second - open/close/unlink 1687 (time 1788738024.88 total 10.00 last 168.64) - open/close/unlink 3269 (time 1788738034.88 total 20.01 last 158.17) - open/close/unlink 4972 (time 1788738044.89 total 30.01 last 170.21) - open/close/unlink 6748 (time 1788738054.89 total 40.01 last 177.57) - open/close/unlink 8555 (time 1788738064.89 total 50.01 last 180.69) - open/close/unlink 10000 (time 1788738072.82 total 57.94 last 182.20) - open/close/unlink 11714 (time 1788738082.84 total 67.96 last 171.19) - open/close/unlink 13504 (time 1788738092.84 total 77.96 last 178.99) - open/close/unlink 15269 (time 1788738102.84 total 87.96 last 176.47) - open/close/unlink 16980 (time 1788738112.84 total 97.96 last 171.04) - open/close/unlink 18736 (time 1788738122.84 total 107.96 last 175.56) - open/close/unlink 20000 (time 1788738130.07 total 115.19 last 174.91) - open/close/unlink 21801 (time 1788738140.08 total 125.20 last 179.98) - open/close/unlink 23608 (time 1788738150.08 total 135.20 last 180.68) - open/close/unlink 25523 (time 1788738160.10 total 145.22 last 191.16) - open/close/unlink 27432 (time 1788738170.10 total 155.23 last 190.73) - open/close/unlink 29315 (time 1788738180.11 total 165.23 last 188.29) - open/close/unlink 30000 (time 1788738183.81 total 168.93 last 184.78) - open/close/unlink 31877 (time 1788738193.81 total 178.93 last 187.67) - open/close/unlink 33688 (time 1788738203.81 total 188.93 last 181.10) - open/close/unlink 35569 (time 1788738213.81 total 198.93 last 188.10) - open/close/unlink 37473 (time 1788738223.81 total 208.93 last 190.39) - open/close/unlink 39378 (time 1788738233.82 total 218.94 last 190.47) - open/close/unlink 40000 (time 1788738237.23 total 222.35 last 181.98) - open/close/unlink 41765 (time 1788738247.24 total 232.36 last 176.34) - open/close/unlink 43526 (time 1788738257.27 total 242.39 last 175.56) - open/close/unlink 45263 (time 1788738267.28 total 252.40 last 173.65) - open/close/unlink 47189 (time 1788738277.28 total 262.40 last 192.57) - open/close/unlink 49111 (time 1788738287.28 total 272.40 last 192.11) - open/close/unlink 50000 (time 1788738293.46 total 278.58 last 143.99) - open/close/unlink 51879 (time 1788738303.46 total 288.58 last 187.76) - open/close/unlink 53765 (time 1788738313.47 total 298.59 last 188.53) - open/close/unlink 55606 (time 1788738323.47 total 308.59 last 184.04) - open/close/unlink 57461 (time 1788738333.47 total 318.59 last 185.44) - open/close/unlink 59320 (time 1788738343.50 total 328.62 last 185.36) - open/close/unlink 60000 (time 1788738347.22 total 332.34 last 182.76) - open/close/unlink 61884 (time 1788738357.23 total 342.35 last 188.33) - open/close/unlink 63747 (time 1788738367.23 total 352.35 last 186.23) total: 64768 open/close/unlink in 358.03 seconds: 180.90 ops/second - open/close/unlink 1685 (time 1788738024.84 total 10.01 last 168.39) - open/close/unlink 3263 (time 1788738034.84 total 20.01 last 157.78) - open/close/unlink 4945 (time 1788738044.84 total 30.01 last 168.13) - open/close/unlink 6725 (time 1788738054.84 total 40.01 last 178.00) - open/close/unlink 8545 (time 1788738064.85 total 50.02 last 181.90) - open/close/unlink 10000 (time 1788738072.90 total 58.07 last 180.73) - open/close/unlink 11717 (time 1788738082.90 total 68.07 last 171.61) - open/close/unlink 13505 (time 1788738092.91 total 78.07 last 178.79) - open/close/unlink 15258 (time 1788738102.91 total 88.08 last 175.18) - open/close/unlink 16970 (time 1788738112.91 total 98.08 last 171.16) - open/close/unlink 18734 (time 1788738122.92 total 108.09 last 176.31) - open/close/unlink 20000 (time 1788738130.22 total 115.39 last 173.32) - open/close/unlink 21825 (time 1788738140.23 total 125.39 last 182.47) - open/close/unlink 23622 (time 1788738150.23 total 135.40 last 179.68) - open/close/unlink 25523 (time 1788738160.23 total 145.40 last 190.07) - open/close/unlink 27426 (time 1788738170.24 total 155.40 last 190.17) - open/close/unlink 29327 (time 1788738180.24 total 165.41 last 190.00) - open/close/unlink 30000 (time 1788738183.79 total 168.96 last 189.44) - open/close/unlink 31849 (time 1788738193.80 total 178.97 last 184.77) - open/close/unlink 33670 (time 1788738203.80 total 188.97 last 182.07) - open/close/unlink 35555 (time 1788738213.81 total 198.98 last 188.42) - open/close/unlink 37456 (time 1788738223.81 total 208.98 last 190.05) - open/close/unlink 39373 (time 1788738233.81 total 218.98 last 191.66) - open/close/unlink 40000 (time 1788738237.33 total 222.50 last 178.26) - open/close/unlink 41722 (time 1788738247.34 total 232.51 last 172.04) - open/close/unlink 43479 (time 1788738257.35 total 242.51 last 175.57) - open/close/unlink 45212 (time 1788738267.35 total 252.52 last 173.25) - open/close/unlink 47138 (time 1788738277.35 total 262.52 last 192.54) - open/close/unlink 49076 (time 1788738287.36 total 272.53 last 193.56) - open/close/unlink 50000 (time 1788738293.59 total 278.76 last 148.45) - open/close/unlink 51870 (time 1788738303.59 total 288.76 last 186.96) - open/close/unlink 53759 (time 1788738313.59 total 298.76 last 188.87) - open/close/unlink 55613 (time 1788738323.60 total 308.77 last 185.18) - open/close/unlink 57449 (time 1788738333.62 total 318.78 last 183.38) - open/close/unlink 59326 (time 1788738343.62 total 328.79 last 187.70) - open/close/unlink 60000 (time 1788738347.34 total 332.51 last 180.93) - open/close/unlink 61889 (time 1788738357.34 total 342.51 last 188.86) - open/close/unlink 63757 (time 1788738367.34 total 352.51 last 186.78) total: 64768 open/close/unlink in 358.24 seconds: 180.79 ops/second sleep 5 for ZFS MDS sleep 5 for ZFS OST Waiting for MDT destroys to complete oleg357-server: cannot open 'lustre-ost2': no such pool pdsh@oleg357-client: oleg357-server: ssh exited with exit code 255 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg357-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg357-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg357-server stop mds service on oleg357-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg357-server unloading modules via unload_modules_local on: 'oleg357-server' oleg357-server: oleg357-server.virtnet: executing unload_modules_local oleg357-server: kmemleak disabled modules unloaded. pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-client: ssh exited with exit code 2 pdsh@oleg357-client: oleg357-server: ssh exited with exit code 2