== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:44:37 (1779126277) start mds service on oleg419-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg419-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg419-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg419-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg419-server: oleg419-server.virtnet: executing set_default_debug -1 all pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 Started lustre-OST0000 oleg419-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg419-client.virtnet: -o user_xattr,flock 192.168.204.119@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1995 (time 1779126311.68 total 10.00 last 199.46) - open/close/unlink 3945 (time 1779126321.68 total 20.00 last 194.95) - open/close/unlink 6138 (time 1779126331.68 total 30.01 last 219.24) - open/close/unlink 8091 (time 1779126341.68 total 40.01 last 195.23) - open/close/unlink 10000 (time 1779126350.31 total 48.63 last 221.42) - open/close/unlink 12179 (time 1779126360.31 total 58.63 last 217.84) - open/close/unlink 14334 (time 1779126370.31 total 68.64 last 215.44) - open/close/unlink 16325 (time 1779126380.31 total 78.64 last 199.03) - open/close/unlink 18501 (time 1779126390.32 total 88.65 last 217.50) - open/close/unlink 20000 (time 1779126397.26 total 95.58 last 216.12) - open/close/unlink 21763 (time 1779126407.26 total 105.59 last 176.15) - open/close/unlink 23336 (time 1779126417.27 total 115.60 last 157.19) - open/close/unlink 24935 (time 1779126427.27 total 125.60 last 159.87) - open/close/unlink 26687 (time 1779126437.28 total 135.60 last 175.14) - open/close/unlink 28412 (time 1779126447.28 total 145.61 last 172.46) - open/close/unlink 30000 (time 1779126455.34 total 153.67 last 196.94) - open/close/unlink 31705 (time 1779126465.34 total 163.67 last 170.46) - open/close/unlink 33477 (time 1779126475.35 total 173.67 last 177.17) - open/close/unlink 35232 (time 1779126485.35 total 183.67 last 175.49) - open/close/unlink 36653 (time 1779126495.35 total 193.68 last 142.02) - open/close/unlink 38241 (time 1779126505.35 total 203.68 last 158.80) - open/close/unlink 40000 (time 1779126514.48 total 212.80 last 192.82) - open/close/unlink 41644 (time 1779126524.48 total 222.80 last 164.35) - open/close/unlink 43583 (time 1779126534.48 total 232.81 last 193.89) - open/close/unlink 45350 (time 1779126544.48 total 242.81 last 176.61) - open/close/unlink 47005 (time 1779126554.49 total 252.81 last 165.46) - open/close/unlink 48551 (time 1779126564.49 total 262.82 last 154.51) - open/close/unlink 49988 (time 1779126574.50 total 272.82 last 143.64) - open/close/unlink 51456 (time 1779126584.50 total 282.82 last 146.79) - open/close/unlink 52985 (time 1779126594.50 total 292.83 last 152.83) - open/close/unlink 54810 (time 1779126604.50 total 302.83 last 182.46) - open/close/unlink 56599 (time 1779126614.51 total 312.83 last 178.86) - open/close/unlink 58604 (time 1779126624.51 total 322.84 last 200.41) - open/close/unlink 60000 (time 1779126632.78 total 331.11 last 168.80) - open/close/unlink 61630 (time 1779126642.79 total 341.11 last 162.92) - open/close/unlink 63480 (time 1779126652.79 total 351.11 last 184.96) total: 64768 open/close/unlink in 358.28 seconds: 180.77 ops/second - open/close/unlink 1936 (time 1779126311.93 total 10.01 last 193.49) - open/close/unlink 3922 (time 1779126321.93 total 20.01 last 198.54) - open/close/unlink 6133 (time 1779126331.93 total 30.01 last 221.03) - open/close/unlink 8083 (time 1779126341.94 total 40.02 last 194.83) - open/close/unlink 10000 (time 1779126350.53 total 48.61 last 223.13) - open/close/unlink 12201 (time 1779126360.53 total 58.61 last 220.08) - open/close/unlink 14343 (time 1779126370.53 total 68.61 last 214.20) - open/close/unlink 16365 (time 1779126380.54 total 78.62 last 202.12) - open/close/unlink 18514 (time 1779126390.54 total 88.62 last 214.87) - open/close/unlink 20000 (time 1779126397.36 total 95.44 last 217.77) - open/close/unlink 21744 (time 1779126407.36 total 105.44 last 174.38) - open/close/unlink 23317 (time 1779126417.37 total 115.45 last 157.23) - open/close/unlink 24921 (time 1779126427.37 total 125.45 last 160.37) - open/close/unlink 26671 (time 1779126437.37 total 135.45 last 174.96) - open/close/unlink 28385 (time 1779126447.37 total 145.45 last 171.39) - open/close/unlink 30000 (time 1779126455.60 total 153.68 last 196.34) - open/close/unlink 31700 (time 1779126465.60 total 163.68 last 169.96) - open/close/unlink 33470 (time 1779126475.61 total 173.69 last 176.91) - open/close/unlink 35225 (time 1779126485.61 total 183.69 last 175.49) - open/close/unlink 36652 (time 1779126495.61 total 193.69 last 142.66) - open/close/unlink 38272 (time 1779126505.61 total 203.69 last 161.96) - open/close/unlink 40000 (time 1779126514.63 total 212.71 last 191.53) - open/close/unlink 41633 (time 1779126524.63 total 222.71 last 163.30) - open/close/unlink 43540 (time 1779126534.64 total 232.72 last 190.65) - open/close/unlink 45317 (time 1779126544.64 total 242.72 last 177.65) - open/close/unlink 46979 (time 1779126554.64 total 252.72 last 166.20) - open/close/unlink 48528 (time 1779126564.64 total 262.72 last 154.90) - open/close/unlink 49970 (time 1779126574.65 total 272.73 last 144.11) - open/close/unlink 51441 (time 1779126584.65 total 282.73 last 147.06) - open/close/unlink 52968 (time 1779126594.65 total 292.73 last 152.67) - open/close/unlink 54793 (time 1779126604.65 total 302.73 last 182.43) - open/close/unlink 56576 (time 1779126614.66 total 312.74 last 178.20) - open/close/unlink 58546 (time 1779126624.66 total 322.74 last 196.92) - open/close/unlink 60000 (time 1779126633.20 total 331.28 last 170.31) - open/close/unlink 61634 (time 1779126643.20 total 341.28 last 163.34) - open/close/unlink 63503 (time 1779126653.20 total 351.28 last 186.90) total: 64768 open/close/unlink in 358.76 seconds: 180.53 ops/second - open/close/unlink 1982 (time 1779126311.92 total 10.00 last 198.11) - open/close/unlink 3965 (time 1779126321.92 total 20.01 last 198.23) - open/close/unlink 6175 (time 1779126331.93 total 30.01 last 220.96) - open/close/unlink 8100 (time 1779126341.93 total 40.01 last 192.46) - open/close/unlink 10000 (time 1779126350.38 total 48.47 last 224.72) - open/close/unlink 12127 (time 1779126360.38 total 58.47 last 212.68) - open/close/unlink 14260 (time 1779126370.39 total 68.47 last 213.19) - open/close/unlink 16287 (time 1779126380.39 total 78.47 last 202.65) - open/close/unlink 18451 (time 1779126390.40 total 88.48 last 216.32) - open/close/unlink 20000 (time 1779126397.53 total 95.61 last 217.15) - open/close/unlink 21752 (time 1779126407.54 total 105.62 last 175.08) - open/close/unlink 23327 (time 1779126417.54 total 115.62 last 157.42) - open/close/unlink 24945 (time 1779126427.54 total 125.62 last 161.78) - open/close/unlink 26688 (time 1779126437.54 total 135.63 last 174.30) - open/close/unlink 28388 (time 1779126447.55 total 145.63 last 169.90) - open/close/unlink 30000 (time 1779126455.71 total 153.79 last 197.50) - open/close/unlink 31718 (time 1779126465.71 total 163.79 last 171.80) - open/close/unlink 33481 (time 1779126475.72 total 173.80 last 176.22) - open/close/unlink 35242 (time 1779126485.72 total 183.80 last 176.04) - open/close/unlink 36647 (time 1779126495.73 total 193.81 last 140.40) - open/close/unlink 38253 (time 1779126505.73 total 203.81 last 160.53) - open/close/unlink 40000 (time 1779126514.80 total 212.88 last 192.67) - open/close/unlink 41648 (time 1779126524.80 total 222.88 last 164.78) - open/close/unlink 43561 (time 1779126534.80 total 232.88 last 191.26) - open/close/unlink 45343 (time 1779126544.80 total 242.89 last 178.16) - open/close/unlink 46976 (time 1779126554.80 total 252.89 last 163.27) - open/close/unlink 48510 (time 1779126564.81 total 262.89 last 153.39) - open/close/unlink 49956 (time 1779126574.81 total 272.89 last 144.51) - open/close/unlink 51449 (time 1779126584.81 total 282.90 last 149.29) - open/close/unlink 52969 (time 1779126594.82 total 292.90 last 151.87) - open/close/unlink 54806 (time 1779126604.82 total 302.90 last 183.68) - open/close/unlink 56593 (time 1779126614.82 total 312.90 last 178.69) - open/close/unlink 58586 (time 1779126624.83 total 322.91 last 199.20) - open/close/unlink 60000 (time 1779126633.19 total 331.27 last 169.14) - open/close/unlink 61624 (time 1779126643.19 total 341.27 last 162.36) - open/close/unlink 63492 (time 1779126653.19 total 351.28 last 186.74) total: 64768 open/close/unlink in 359.19 seconds: 180.32 ops/second - open/close/unlink 1953 (time 1779126311.95 total 10.00 last 195.22) - open/close/unlink 3926 (time 1779126321.95 total 20.00 last 197.29) - open/close/unlink 6117 (time 1779126331.95 total 30.01 last 219.08) - open/close/unlink 8066 (time 1779126341.95 total 40.01 last 194.85) - open/close/unlink 10000 (time 1779126350.60 total 48.65 last 223.73) - open/close/unlink 12185 (time 1779126360.60 total 58.66 last 218.43) - open/close/unlink 14308 (time 1779126370.60 total 68.66 last 212.28) - open/close/unlink 16274 (time 1779126380.60 total 78.66 last 196.54) - open/close/unlink 18430 (time 1779126390.61 total 88.66 last 215.56) - open/close/unlink 20000 (time 1779126397.70 total 95.75 last 221.42) - open/close/unlink 21687 (time 1779126407.70 total 105.76 last 168.60) - open/close/unlink 23261 (time 1779126417.70 total 115.76 last 157.37) - open/close/unlink 24892 (time 1779126427.71 total 125.76 last 163.05) - open/close/unlink 26637 (time 1779126437.71 total 135.77 last 174.38) - open/close/unlink 28379 (time 1779126447.72 total 145.77 last 174.15) - open/close/unlink 30000 (time 1779126455.96 total 154.02 last 196.67) - open/close/unlink 31700 (time 1779126465.96 total 164.02 last 170.00) - open/close/unlink 33455 (time 1779126475.97 total 174.02 last 175.37) - open/close/unlink 35203 (time 1779126485.97 total 184.03 last 174.74) - open/close/unlink 36610 (time 1779126495.97 total 194.03 last 140.66) - open/close/unlink 38220 (time 1779126505.97 total 204.03 last 160.98) - open/close/unlink 40000 (time 1779126515.25 total 213.31 last 191.84) - open/close/unlink 41639 (time 1779126525.25 total 223.31 last 163.88) - open/close/unlink 43552 (time 1779126535.26 total 233.31 last 191.23) - open/close/unlink 45333 (time 1779126545.26 total 243.31 last 178.08) - open/close/unlink 46967 (time 1779126555.26 total 253.32 last 163.34) - open/close/unlink 48481 (time 1779126565.26 total 263.32 last 151.36) - open/close/unlink 49918 (time 1779126575.27 total 273.33 last 143.61) - open/close/unlink 51421 (time 1779126585.27 total 283.33 last 150.27) - open/close/unlink 52921 (time 1779126595.27 total 293.33 last 149.97) - open/close/unlink 54763 (time 1779126605.28 total 303.34 last 184.10) - open/close/unlink 56595 (time 1779126615.28 total 313.34 last 183.19) - open/close/unlink 58553 (time 1779126625.28 total 323.34 last 195.77) - open/close/unlink 60000 (time 1779126633.95 total 332.01 last 166.96) - open/close/unlink 61628 (time 1779126643.95 total 342.01 last 162.75) - open/close/unlink 63549 (time 1779126653.96 total 352.01 last 192.00) total: 64768 open/close/unlink in 361.37 seconds: 179.23 ops/second - open/close/unlink 1942 (time 1779126311.99 total 10.00 last 194.18) - open/close/unlink 3904 (time 1779126321.99 total 20.00 last 196.16) - open/close/unlink 6098 (time 1779126331.99 total 30.00 last 219.38) - open/close/unlink 8018 (time 1779126342.00 total 40.01 last 191.97) - open/close/unlink 10000 (time 1779126350.94 total 48.95 last 221.49) - open/close/unlink 12142 (time 1779126360.95 total 58.96 last 214.13) - open/close/unlink 14278 (time 1779126370.95 total 68.96 last 213.50) - open/close/unlink 16302 (time 1779126380.96 total 78.97 last 202.35) - open/close/unlink 18461 (time 1779126390.96 total 88.97 last 215.87) - open/close/unlink 20000 (time 1779126397.97 total 95.98 last 219.33) - open/close/unlink 21680 (time 1779126407.97 total 105.98 last 168.00) - open/close/unlink 23261 (time 1779126417.98 total 115.99 last 158.04) - open/close/unlink 24902 (time 1779126427.98 total 125.99 last 164.06) - open/close/unlink 26637 (time 1779126438.02 total 136.03 last 172.78) - open/close/unlink 28348 (time 1779126448.03 total 146.04 last 171.01) - open/close/unlink 30000 (time 1779126456.42 total 154.43 last 196.92) - open/close/unlink 31698 (time 1779126466.42 total 164.43 last 169.77) - open/close/unlink 33470 (time 1779126476.42 total 174.43 last 177.14) - open/close/unlink 35214 (time 1779126486.43 total 184.44 last 174.33) - open/close/unlink 36619 (time 1779126496.43 total 194.44 last 140.45) - open/close/unlink 38281 (time 1779126506.43 total 204.44 last 166.16) - open/close/unlink 40000 (time 1779126515.36 total 213.37 last 192.54) - open/close/unlink 41641 (time 1779126525.36 total 223.37 last 164.03) - open/close/unlink 43575 (time 1779126535.36 total 233.37 last 193.40) - open/close/unlink 45353 (time 1779126545.37 total 243.38 last 177.77) - open/close/unlink 46971 (time 1779126555.37 total 253.38 last 161.79) - open/close/unlink 48502 (time 1779126565.37 total 263.38 last 153.05) - open/close/unlink 49918 (time 1779126575.38 total 273.39 last 141.47) - open/close/unlink 51429 (time 1779126585.38 total 283.39 last 151.08) - open/close/unlink 52908 (time 1779126595.38 total 293.39 last 147.87) - open/close/unlink 54742 (time 1779126605.39 total 303.40 last 183.34) - open/close/unlink 56577 (time 1779126615.39 total 313.40 last 183.48) - open/close/unlink 58521 (time 1779126625.39 total 323.40 last 194.35) - open/close/unlink 60000 (time 1779126634.24 total 332.25 last 167.13) - open/close/unlink 61629 (time 1779126644.24 total 342.25 last 162.87) - open/close/unlink 63556 (time 1779126654.24 total 352.25 last 192.65) total: 64768 open/close/unlink in 361.78 seconds: 179.03 ops/second Waiting for MDT destroys to complete oleg419-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg419-client: oleg419-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg419-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg419-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg419-server stop mds service on oleg419-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg419-server unloading modules via unload_modules_local on: 'oleg419-server' oleg419-server: oleg419-server.virtnet: executing unload_modules_local oleg419-server: modules unloaded. pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-client: ssh exited with exit code 2 pdsh@oleg419-client: oleg419-server: ssh exited with exit code 2