== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 08:48:24 (1788353304) start mds service on oleg207-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg207-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 start ost1 service on oleg207-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg207-server: oleg207-server.virtnet: executing set_default_debug -1 all pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 Started lustre-OST0000 oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 mount lustre on /mnt/lustre..... Starting client: oleg207-client.virtnet: -o user_xattr,flock 192.168.202.107@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 2031 (time 1788353346.98 total 10.00 last 203.02) - open/close/unlink 4207 (time 1788353356.98 total 20.00 last 217.59) - open/close/unlink 6505 (time 1788353366.98 total 30.01 last 229.72) - open/close/unlink 8800 (time 1788353376.99 total 40.01 last 229.42) - open/close/unlink 10000 (time 1788353382.14 total 45.17 last 232.57) - open/close/unlink 12211 (time 1788353392.15 total 55.17 last 221.04) - open/close/unlink 14548 (time 1788353402.15 total 65.18 last 233.63) - open/close/unlink 16998 (time 1788353412.15 total 75.18 last 244.99) - open/close/unlink 19435 (time 1788353422.15 total 85.18 last 243.69) - open/close/unlink 20000 (time 1788353424.41 total 87.44 last 250.16) - open/close/unlink 22323 (time 1788353434.41 total 97.44 last 232.23) - open/close/unlink 24670 (time 1788353444.42 total 107.44 last 234.64) - open/close/unlink 27009 (time 1788353454.42 total 117.44 last 233.84) - open/close/unlink 29317 (time 1788353464.42 total 127.45 last 230.77) - open/close/unlink 30000 (time 1788353467.28 total 130.30 last 238.98) - open/close/unlink 32328 (time 1788353477.28 total 140.30 last 232.79) - open/close/unlink 34608 (time 1788353487.28 total 150.31 last 227.98) - open/close/unlink 37015 (time 1788353497.28 total 160.31 last 240.67) - open/close/unlink 39304 (time 1788353507.28 total 170.31 last 228.88) - open/close/unlink 40000 (time 1788353510.06 total 173.09 last 250.21) - open/close/unlink 42294 (time 1788353520.06 total 183.09 last 229.40) - open/close/unlink 44671 (time 1788353530.06 total 193.09 last 237.66) - open/close/unlink 47084 (time 1788353540.07 total 203.09 last 241.22) - open/close/unlink 49508 (time 1788353550.07 total 213.10 last 242.38) - open/close/unlink 51921 (time 1788353560.07 total 223.10 last 241.23) - open/close/unlink 54317 (time 1788353570.07 total 233.10 last 239.56) - open/close/unlink 56635 (time 1788353580.08 total 243.10 last 231.74) - open/close/unlink 59020 (time 1788353590.08 total 253.10 last 238.48) - open/close/unlink 60000 (time 1788353594.20 total 257.23 last 237.49) - open/close/unlink 62409 (time 1788353604.21 total 267.23 last 240.84) total: 64768 open/close/unlink in 276.88 seconds: 233.92 ops/second - open/close/unlink 2038 (time 1788353347.00 total 10.00 last 203.78) - open/close/unlink 4213 (time 1788353357.00 total 20.00 last 217.50) - open/close/unlink 6535 (time 1788353367.01 total 30.00 last 232.16) - open/close/unlink 8839 (time 1788353377.01 total 40.00 last 230.38) - open/close/unlink 10000 (time 1788353382.07 total 45.06 last 229.43) - open/close/unlink 12235 (time 1788353392.07 total 55.07 last 223.46) - open/close/unlink 14526 (time 1788353402.07 total 65.07 last 229.06) - open/close/unlink 17016 (time 1788353412.07 total 75.07 last 248.99) - open/close/unlink 19387 (time 1788353422.07 total 85.07 last 237.02) - open/close/unlink 20000 (time 1788353424.48 total 87.47 last 255.06) - open/close/unlink 22302 (time 1788353434.48 total 97.48 last 230.14) - open/close/unlink 24613 (time 1788353444.48 total 107.48 last 231.09) - open/close/unlink 26918 (time 1788353454.48 total 117.48 last 230.49) - open/close/unlink 29206 (time 1788353464.48 total 127.48 last 228.78) - open/close/unlink 30000 (time 1788353467.75 total 130.75 last 242.87) - open/close/unlink 32317 (time 1788353477.76 total 140.75 last 231.57) - open/close/unlink 34604 (time 1788353487.80 total 150.80 last 227.76) - open/close/unlink 37054 (time 1788353497.80 total 160.80 last 245.00) - open/close/unlink 39333 (time 1788353507.80 total 170.80 last 227.86) - open/close/unlink 40000 (time 1788353510.52 total 173.52 last 245.20) - open/close/unlink 42305 (time 1788353520.52 total 183.52 last 230.48) - open/close/unlink 44673 (time 1788353530.52 total 193.52 last 236.76) - open/close/unlink 47104 (time 1788353540.53 total 203.52 last 243.00) - open/close/unlink 49529 (time 1788353550.53 total 213.53 last 242.36) - open/close/unlink 51921 (time 1788353560.53 total 223.53 last 239.14) - open/close/unlink 54364 (time 1788353570.54 total 233.53 last 244.27) - open/close/unlink 56697 (time 1788353580.54 total 243.53 last 233.30) - open/close/unlink 59051 (time 1788353590.54 total 253.54 last 235.32) - open/close/unlink 60000 (time 1788353594.60 total 257.60 last 233.51) - open/close/unlink 62418 (time 1788353604.60 total 267.60 last 241.77) - open/close/unlink 64702 (time 1788353614.62 total 277.61 last 228.14) total: 64768 open/close/unlink in 279.36 seconds: 231.85 ops/second - open/close/unlink 2039 (time 1788353346.69 total 10.00 last 203.83) - open/close/unlink 4192 (time 1788353356.70 total 20.01 last 215.21) - open/close/unlink 6468 (time 1788353366.70 total 30.01 last 227.57) - open/close/unlink 8777 (time 1788353376.70 total 40.01 last 230.86) - open/close/unlink 10000 (time 1788353382.07 total 45.38 last 227.82) - open/close/unlink 12263 (time 1788353392.07 total 55.38 last 226.26) - open/close/unlink 14586 (time 1788353402.07 total 65.38 last 232.26) - open/close/unlink 17073 (time 1788353412.07 total 75.38 last 248.67) - open/close/unlink 19460 (time 1788353422.07 total 85.38 last 238.70) - open/close/unlink 20000 (time 1788353424.30 total 87.61 last 242.38) - open/close/unlink 22300 (time 1788353434.30 total 97.61 last 229.96) - open/close/unlink 24639 (time 1788353444.30 total 107.61 last 233.89) - open/close/unlink 26947 (time 1788353454.31 total 117.62 last 230.73) - open/close/unlink 29235 (time 1788353464.31 total 127.62 last 228.78) - open/close/unlink 30000 (time 1788353467.45 total 130.76 last 243.58) - open/close/unlink 32271 (time 1788353477.45 total 140.76 last 227.05) - open/close/unlink 34557 (time 1788353487.45 total 150.76 last 228.59) - open/close/unlink 36926 (time 1788353497.45 total 160.76 last 236.85) - open/close/unlink 39237 (time 1788353507.46 total 170.77 last 231.03) - open/close/unlink 40000 (time 1788353510.64 total 173.95 last 239.88) - open/close/unlink 42284 (time 1788353520.64 total 183.95 last 228.31) - open/close/unlink 44655 (time 1788353530.64 total 193.95 last 237.09) - open/close/unlink 47072 (time 1788353540.64 total 203.95 last 241.69) - open/close/unlink 49523 (time 1788353550.65 total 213.96 last 245.00) - open/close/unlink 51961 (time 1788353560.65 total 223.96 last 243.80) - open/close/unlink 54351 (time 1788353570.65 total 233.96 last 238.99) - open/close/unlink 56683 (time 1788353580.65 total 243.96 last 233.19) - open/close/unlink 59052 (time 1788353590.65 total 253.96 last 236.76) - open/close/unlink 60000 (time 1788353594.69 total 258.00 last 234.68) - open/close/unlink 62432 (time 1788353604.70 total 268.01 last 243.06) - open/close/unlink 64681 (time 1788353614.72 total 278.03 last 224.41) total: 64768 open/close/unlink in 280.10 seconds: 231.23 ops/second - open/close/unlink 2039 (time 1788353346.95 total 10.00 last 203.90) - open/close/unlink 4202 (time 1788353356.95 total 20.00 last 216.24) - open/close/unlink 6506 (time 1788353366.95 total 30.00 last 230.38) - open/close/unlink 8794 (time 1788353376.95 total 40.01 last 228.77) - open/close/unlink 10000 (time 1788353382.07 total 45.12 last 235.55) - open/close/unlink 12216 (time 1788353392.08 total 55.13 last 221.50) - open/close/unlink 14513 (time 1788353402.08 total 65.13 last 229.70) - open/close/unlink 16950 (time 1788353412.08 total 75.13 last 243.65) - open/close/unlink 19377 (time 1788353422.08 total 85.13 last 242.65) - open/close/unlink 20000 (time 1788353424.60 total 87.65 last 247.12) - open/close/unlink 22304 (time 1788353434.60 total 97.66 last 230.38) - open/close/unlink 24633 (time 1788353444.60 total 107.66 last 232.86) - open/close/unlink 26927 (time 1788353454.61 total 117.66 last 229.31) - open/close/unlink 29211 (time 1788353464.61 total 127.66 last 228.39) - open/close/unlink 30000 (time 1788353467.93 total 130.98 last 237.45) - open/close/unlink 32332 (time 1788353477.94 total 140.99 last 233.00) - open/close/unlink 34609 (time 1788353487.94 total 151.00 last 227.66) - open/close/unlink 37015 (time 1788353497.95 total 161.00 last 240.47) - open/close/unlink 39308 (time 1788353507.95 total 171.00 last 229.22) - open/close/unlink 40000 (time 1788353510.80 total 173.85 last 242.96) - open/close/unlink 42337 (time 1788353520.80 total 183.86 last 233.63) - open/close/unlink 44684 (time 1788353530.81 total 193.86 last 234.59) - open/close/unlink 47120 (time 1788353540.81 total 203.86 last 243.59) - open/close/unlink 49576 (time 1788353550.81 total 213.86 last 245.53) - open/close/unlink 52004 (time 1788353560.81 total 223.87 last 242.70) - open/close/unlink 54377 (time 1788353570.82 total 233.87 last 237.20) - open/close/unlink 56685 (time 1788353580.82 total 243.87 last 230.79) - open/close/unlink 59025 (time 1788353590.82 total 253.88 last 233.91) - open/close/unlink 60000 (time 1788353594.93 total 257.98 last 237.40) - open/close/unlink 62452 (time 1788353604.93 total 267.99 last 245.06) - open/close/unlink 64658 (time 1788353614.94 total 278.00 last 220.42) total: 64768 open/close/unlink in 280.40 seconds: 230.99 ops/second - open/close/unlink 2045 (time 1788353346.98 total 10.00 last 204.46) - open/close/unlink 4227 (time 1788353356.98 total 20.00 last 218.18) - open/close/unlink 6557 (time 1788353366.98 total 30.00 last 232.96) - open/close/unlink 8848 (time 1788353376.98 total 40.00 last 229.10) - open/close/unlink 10000 (time 1788353381.96 total 44.98 last 231.57) - open/close/unlink 12208 (time 1788353391.96 total 54.98 last 220.80) - open/close/unlink 14519 (time 1788353401.96 total 64.98 last 231.07) - open/close/unlink 16967 (time 1788353411.96 total 74.98 last 244.74) - open/close/unlink 19394 (time 1788353421.96 total 84.99 last 242.66) - open/close/unlink 20000 (time 1788353424.48 total 87.50 last 240.68) - open/close/unlink 22277 (time 1788353434.48 total 97.50 last 227.70) - open/close/unlink 24600 (time 1788353444.48 total 107.50 last 232.30) - open/close/unlink 26913 (time 1788353454.48 total 117.50 last 231.30) - open/close/unlink 29200 (time 1788353464.48 total 127.50 last 228.69) - open/close/unlink 30000 (time 1788353467.77 total 130.79 last 243.17) - open/close/unlink 32290 (time 1788353477.77 total 140.80 last 228.92) - open/close/unlink 34569 (time 1788353487.80 total 150.82 last 227.37) - open/close/unlink 36990 (time 1788353497.80 total 160.82 last 242.03) - open/close/unlink 39262 (time 1788353507.80 total 170.83 last 227.12) - open/close/unlink 40000 (time 1788353510.80 total 173.82 last 246.49) - open/close/unlink 42294 (time 1788353520.80 total 183.82 last 229.40) - open/close/unlink 44660 (time 1788353530.80 total 193.82 last 236.57) - open/close/unlink 47059 (time 1788353540.80 total 203.82 last 239.90) - open/close/unlink 49504 (time 1788353550.80 total 213.82 last 244.44) - open/close/unlink 50000 (time 1788353552.83 total 215.85 last 244.63) - open/close/unlink 52411 (time 1788353562.83 total 225.86 last 241.00) - open/close/unlink 54757 (time 1788353572.84 total 235.86 last 234.57) - open/close/unlink 57125 (time 1788353582.84 total 245.86 last 236.73) - open/close/unlink 59499 (time 1788353592.84 total 255.86 last 237.37) - open/close/unlink 60000 (time 1788353595.05 total 258.07 last 226.45) - open/close/unlink 62439 (time 1788353605.06 total 268.08 last 243.74) - open/close/unlink 64632 (time 1788353615.07 total 278.09 last 219.12) total: 64768 open/close/unlink in 280.65 seconds: 230.78 ops/second Waiting for MDT destroys to complete oleg207-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg207-client: oleg207-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg207-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg207-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg207-server stop mds service on oleg207-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg207-server unloading modules via unload_modules_local on: 'oleg207-server' oleg207-server: oleg207-server.virtnet: executing unload_modules_local oleg207-server: modules unloaded. pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-client: ssh exited with exit code 2 pdsh@oleg207-client: oleg207-server: ssh exited with exit code 2