== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 13:45:23 (1782927923) start mds service on oleg434-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg434-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg434-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg434-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg434-server: oleg434-server.virtnet: executing set_default_debug -1 all pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 Started lustre-OST0000 oleg434-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg434-client.virtnet: -o user_xattr,flock 192.168.204.134@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 1884 (time 1782927968.31 total 10.03 last 187.89) - open/close/unlink 3870 (time 1782927978.31 total 20.03 last 198.55) - open/close/unlink 5931 (time 1782927988.31 total 30.03 last 206.10) - open/close/unlink 8202 (time 1782927998.31 total 40.03 last 227.03) - open/close/unlink 10000 (time 1782928006.67 total 48.39 last 215.17) - open/close/unlink 12273 (time 1782928016.67 total 58.39 last 227.29) - open/close/unlink 14582 (time 1782928026.67 total 68.39 last 230.90) - open/close/unlink 16733 (time 1782928036.67 total 78.39 last 215.01) - open/close/unlink 18833 (time 1782928046.67 total 88.40 last 209.93) - open/close/unlink 20000 (time 1782928051.74 total 93.46 last 230.35) - open/close/unlink 22141 (time 1782928061.74 total 103.46 last 214.08) - open/close/unlink 24457 (time 1782928071.74 total 113.46 last 231.59) - open/close/unlink 26628 (time 1782928081.75 total 123.47 last 217.04) - open/close/unlink 28889 (time 1782928091.75 total 133.47 last 226.08) - open/close/unlink 30000 (time 1782928096.77 total 138.49 last 221.08) - open/close/unlink 32211 (time 1782928106.77 total 148.50 last 221.04) - open/close/unlink 34238 (time 1782928116.78 total 158.50 last 202.66) - open/close/unlink 36371 (time 1782928126.78 total 168.50 last 213.23) - open/close/unlink 38545 (time 1782928136.78 total 178.50 last 217.38) - open/close/unlink 40000 (time 1782928143.50 total 185.22 last 216.45) - open/close/unlink 42219 (time 1782928153.51 total 195.23 last 221.79) - open/close/unlink 44407 (time 1782928163.51 total 205.23 last 218.75) - open/close/unlink 46451 (time 1782928173.51 total 215.23 last 204.36) - open/close/unlink 48370 (time 1782928183.51 total 225.23 last 191.89) - open/close/unlink 50000 (time 1782928191.45 total 233.17 last 205.34) - open/close/unlink 52047 (time 1782928201.45 total 243.18 last 204.64) - open/close/unlink 54065 (time 1782928211.46 total 253.18 last 201.71) - open/close/unlink 56123 (time 1782928221.46 total 263.19 last 205.69) - open/close/unlink 58050 (time 1782928231.47 total 273.19 last 192.61) - open/close/unlink 60000 (time 1782928241.11 total 282.83 last 202.22) - open/close/unlink 62039 (time 1782928251.11 total 292.83 last 203.90) - open/close/unlink 64131 (time 1782928261.11 total 302.83 last 209.17) total: 64768 open/close/unlink in 305.86 seconds: 211.76 ops/second - open/close/unlink 1892 (time 1782927968.07 total 10.00 last 189.17) - open/close/unlink 3869 (time 1782927978.08 total 20.01 last 197.62) - open/close/unlink 5924 (time 1782927988.08 total 30.01 last 205.42) - open/close/unlink 8180 (time 1782927998.08 total 40.01 last 225.56) - open/close/unlink 10000 (time 1782928006.44 total 48.37 last 217.86) - open/close/unlink 12263 (time 1782928016.44 total 58.37 last 226.24) - open/close/unlink 14562 (time 1782928026.44 total 68.37 last 229.84) - open/close/unlink 16704 (time 1782928036.44 total 78.37 last 214.17) - open/close/unlink 18773 (time 1782928046.45 total 88.38 last 206.83) - open/close/unlink 20000 (time 1782928051.71 total 93.63 last 233.35) - open/close/unlink 22114 (time 1782928061.71 total 103.64 last 211.33) - open/close/unlink 24449 (time 1782928071.71 total 113.64 last 233.47) - open/close/unlink 26610 (time 1782928081.71 total 123.64 last 216.08) - open/close/unlink 28890 (time 1782928091.71 total 133.64 last 227.98) - open/close/unlink 30000 (time 1782928096.72 total 138.64 last 221.87) - open/close/unlink 32196 (time 1782928106.72 total 148.64 last 219.56) - open/close/unlink 34294 (time 1782928116.72 total 158.65 last 209.67) - open/close/unlink 36427 (time 1782928126.72 total 168.65 last 213.28) - open/close/unlink 38581 (time 1782928136.73 total 178.65 last 215.37) - open/close/unlink 40000 (time 1782928143.36 total 185.28 last 213.97) - open/close/unlink 42241 (time 1782928153.36 total 195.29 last 224.06) - open/close/unlink 44414 (time 1782928163.36 total 205.29 last 217.20) - open/close/unlink 46489 (time 1782928173.37 total 215.29 last 207.45) - open/close/unlink 48384 (time 1782928183.37 total 225.30 last 189.35) - open/close/unlink 50000 (time 1782928191.25 total 233.17 last 205.27) - open/close/unlink 52043 (time 1782928201.25 total 243.18 last 204.22) - open/close/unlink 54029 (time 1782928211.25 total 253.18 last 198.56) - open/close/unlink 56097 (time 1782928221.25 total 263.18 last 206.79) - open/close/unlink 58017 (time 1782928231.25 total 273.18 last 191.99) - open/close/unlink 60000 (time 1782928241.10 total 283.03 last 201.28) - open/close/unlink 62026 (time 1782928251.11 total 293.04 last 202.52) - open/close/unlink 64093 (time 1782928261.11 total 303.04 last 206.65) total: 64768 open/close/unlink in 306.45 seconds: 211.35 ops/second - open/close/unlink 1881 (time 1782927968.18 total 10.00 last 188.09) - open/close/unlink 3865 (time 1782927978.18 total 20.00 last 198.36) - open/close/unlink 5948 (time 1782927988.18 total 30.01 last 208.25) - open/close/unlink 8206 (time 1782927998.19 total 40.01 last 225.71) - open/close/unlink 10000 (time 1782928006.44 total 48.26 last 217.52) - open/close/unlink 12248 (time 1782928016.44 total 58.26 last 224.75) - open/close/unlink 14578 (time 1782928026.44 total 68.26 last 232.91) - open/close/unlink 16731 (time 1782928036.44 total 78.26 last 215.26) - open/close/unlink 18800 (time 1782928046.45 total 88.27 last 206.83) - open/close/unlink 20000 (time 1782928051.46 total 93.29 last 239.15) - open/close/unlink 22139 (time 1782928061.47 total 103.29 last 213.83) - open/close/unlink 24455 (time 1782928071.48 total 113.30 last 231.42) - open/close/unlink 26616 (time 1782928081.48 total 123.30 last 216.07) - open/close/unlink 28878 (time 1782928091.48 total 133.30 last 226.18) - open/close/unlink 30000 (time 1782928096.48 total 138.30 last 224.45) - open/close/unlink 32192 (time 1782928106.48 total 148.30 last 219.12) - open/close/unlink 34292 (time 1782928116.48 total 158.30 last 209.95) - open/close/unlink 36407 (time 1782928126.48 total 168.31 last 211.48) - open/close/unlink 38557 (time 1782928136.49 total 178.31 last 214.96) - open/close/unlink 40000 (time 1782928143.14 total 184.96 last 216.75) - open/close/unlink 42232 (time 1782928153.15 total 194.97 last 223.14) - open/close/unlink 44394 (time 1782928163.15 total 204.97 last 216.19) - open/close/unlink 46445 (time 1782928173.15 total 214.97 last 205.04) - open/close/unlink 48380 (time 1782928183.15 total 224.97 last 193.46) - open/close/unlink 50000 (time 1782928191.10 total 232.93 last 203.68) - open/close/unlink 52021 (time 1782928201.11 total 242.93 last 201.94) - open/close/unlink 54006 (time 1782928211.11 total 252.93 last 198.50) - open/close/unlink 56082 (time 1782928221.11 total 262.93 last 207.60) - open/close/unlink 58016 (time 1782928231.12 total 272.94 last 193.36) - open/close/unlink 60000 (time 1782928241.06 total 282.88 last 199.51) - open/close/unlink 62010 (time 1782928251.06 total 292.88 last 200.98) - open/close/unlink 64083 (time 1782928261.06 total 302.88 last 207.25) total: 64768 open/close/unlink in 306.38 seconds: 211.40 ops/second - open/close/unlink 1874 (time 1782927968.12 total 10.02 last 187.05) - open/close/unlink 3852 (time 1782927978.13 total 20.02 last 197.69) - open/close/unlink 5920 (time 1782927988.13 total 30.03 last 206.75) - open/close/unlink 8197 (time 1782927998.13 total 40.03 last 227.68) - open/close/unlink 10000 (time 1782928006.44 total 48.34 last 216.97) - open/close/unlink 12291 (time 1782928016.45 total 58.34 last 229.03) - open/close/unlink 14582 (time 1782928026.45 total 68.34 last 229.06) - open/close/unlink 16713 (time 1782928036.45 total 78.35 last 213.06) - open/close/unlink 18799 (time 1782928046.45 total 88.35 last 208.53) - open/close/unlink 20000 (time 1782928051.50 total 93.39 last 238.11) - open/close/unlink 22124 (time 1782928061.50 total 103.39 last 212.40) - open/close/unlink 24458 (time 1782928071.50 total 113.40 last 233.21) - open/close/unlink 26593 (time 1782928081.51 total 123.40 last 213.46) - open/close/unlink 28838 (time 1782928091.51 total 133.40 last 224.48) - open/close/unlink 30000 (time 1782928096.59 total 138.49 last 228.52) - open/close/unlink 32195 (time 1782928106.59 total 148.49 last 219.43) - open/close/unlink 34286 (time 1782928116.60 total 158.49 last 209.09) - open/close/unlink 36383 (time 1782928126.60 total 168.49 last 209.69) - open/close/unlink 38525 (time 1782928136.60 total 178.50 last 214.14) - open/close/unlink 40000 (time 1782928143.47 total 185.36 last 214.81) - open/close/unlink 42220 (time 1782928153.47 total 195.37 last 221.87) - open/close/unlink 44397 (time 1782928163.47 total 205.37 last 217.70) - open/close/unlink 46452 (time 1782928173.48 total 215.38 last 205.35) - open/close/unlink 48359 (time 1782928183.48 total 225.38 last 190.63) - open/close/unlink 50000 (time 1782928191.45 total 233.34 last 206.04) - open/close/unlink 52038 (time 1782928201.45 total 243.35 last 203.73) - open/close/unlink 54009 (time 1782928211.46 total 253.35 last 196.98) - open/close/unlink 56058 (time 1782928221.46 total 263.36 last 204.81) - open/close/unlink 58001 (time 1782928231.46 total 273.36 last 194.22) - open/close/unlink 60000 (time 1782928241.42 total 283.32 last 200.78) - open/close/unlink 62018 (time 1782928251.42 total 293.32 last 201.78) - open/close/unlink 64067 (time 1782928261.42 total 303.32 last 204.90) total: 64768 open/close/unlink in 307.72 seconds: 210.48 ops/second - open/close/unlink 1881 (time 1782927968.17 total 10.00 last 188.06) - open/close/unlink 3846 (time 1782927978.17 total 20.00 last 196.48) - open/close/unlink 5901 (time 1782927988.18 total 30.01 last 205.43) - open/close/unlink 8181 (time 1782927998.18 total 40.01 last 227.90) - open/close/unlink 10000 (time 1782928006.63 total 48.46 last 215.42) - open/close/unlink 12283 (time 1782928016.63 total 58.46 last 228.30) - open/close/unlink 14584 (time 1782928026.63 total 68.46 last 230.08) - open/close/unlink 16744 (time 1782928036.63 total 78.46 last 215.99) - open/close/unlink 18856 (time 1782928046.63 total 88.46 last 211.17) - open/close/unlink 20000 (time 1782928051.47 total 93.30 last 236.11) - open/close/unlink 22139 (time 1782928061.48 total 103.31 last 213.83) - open/close/unlink 24441 (time 1782928071.49 total 113.31 last 230.02) - open/close/unlink 26574 (time 1782928081.49 total 123.32 last 213.22) - open/close/unlink 28810 (time 1782928091.49 total 133.32 last 223.58) - open/close/unlink 30000 (time 1782928096.74 total 138.57 last 226.63) - open/close/unlink 32198 (time 1782928106.74 total 148.57 last 219.75) - open/close/unlink 34241 (time 1782928116.75 total 158.58 last 204.20) - open/close/unlink 36359 (time 1782928126.75 total 168.58 last 211.76) - open/close/unlink 38524 (time 1782928136.76 total 178.58 last 216.39) - open/close/unlink 40000 (time 1782928143.66 total 185.49 last 213.74) - open/close/unlink 42211 (time 1782928153.66 total 195.49 last 221.07) - open/close/unlink 44396 (time 1782928163.66 total 205.49 last 218.46) - open/close/unlink 46449 (time 1782928173.67 total 215.50 last 205.22) - open/close/unlink 48389 (time 1782928183.67 total 225.50 last 193.95) - open/close/unlink 50000 (time 1782928191.54 total 233.36 last 204.85) - open/close/unlink 52022 (time 1782928201.54 total 243.37 last 202.13) - open/close/unlink 54002 (time 1782928211.54 total 253.37 last 197.97) - open/close/unlink 56055 (time 1782928221.54 total 263.37 last 205.22) - open/close/unlink 57983 (time 1782928231.55 total 273.37 last 192.76) - open/close/unlink 59988 (time 1782928241.55 total 283.38 last 200.45) - open/close/unlink 62015 (time 1782928251.55 total 293.38 last 202.68) - open/close/unlink 64082 (time 1782928261.55 total 303.38 last 206.68) total: 64768 open/close/unlink in 307.98 seconds: 210.30 ops/second Waiting for MDT destroys to complete oleg434-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg434-client: oleg434-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg434-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg434-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg434-server stop mds service on oleg434-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg434-server unloading modules via unload_modules_local on: 'oleg434-server' oleg434-server: oleg434-server.virtnet: executing unload_modules_local oleg434-server: modules unloaded. pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-server: ssh exited with exit code 2 pdsh@oleg434-client: oleg434-client: ssh exited with exit code 2