== conf-sanity test 106: check osp llog processing when catalog is wrapped ========================================================== 15:59:49 (1782849589) start mds service on oleg351-server Start mds1: mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0000 start mds service on oleg351-server Start mds2: mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-MDT0001 oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid oleg351-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid start ost1 service on oleg351-server Start ost1: mount -t lustre -o localrecov /dev/mapper/ost1_flakey /mnt/lustre-ost1 seq.cli-lustre-OST0000-super.width=65536 oleg351-server: oleg351-server.virtnet: executing set_default_debug -1 all pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 Started lustre-OST0000 oleg351-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid mount lustre on /mnt/lustre..... Starting client: oleg351-client.virtnet: -o user_xattr,flock 192.168.203.151@tcp:/lustre /mnt/lustre UUID Inodes IUsed IFree IUse% Mounted on lustre-MDT0000_UUID 79992 274 79718 1% /mnt/lustre[MDT:0] lustre-MDT0001_UUID 79992 255 79737 1% /mnt/lustre[MDT:1] lustre-OST0000_UUID 50000 433 49567 1% /mnt/lustre[OST:0] filesystem_summary: 50096 529 49567 2% /mnt/lustre seq.cli-lustre-OST0000-super.width=0x1ffffff fail_loc=0x1312 fail_val=5 - open/close/unlink 2382 (time 1782849633.75 total 10.00 last 238.12) - open/close/unlink 4516 (time 1782849643.76 total 20.01 last 213.36) - open/close/unlink 6692 (time 1782849653.76 total 30.01 last 217.49) - open/close/unlink 8826 (time 1782849663.77 total 40.01 last 213.33) - open/close/unlink 10000 (time 1782849668.86 total 45.11 last 230.45) - open/close/unlink 12197 (time 1782849678.86 total 55.11 last 219.64) - open/close/unlink 14487 (time 1782849688.86 total 65.11 last 228.97) - open/close/unlink 16791 (time 1782849698.87 total 75.11 last 230.35) - open/close/unlink 19051 (time 1782849708.87 total 85.11 last 225.99) - open/close/unlink 20000 (time 1782849712.81 total 89.06 last 240.49) - open/close/unlink 22341 (time 1782849722.81 total 99.06 last 234.06) - open/close/unlink 24574 (time 1782849732.81 total 109.06 last 223.29) - open/close/unlink 26558 (time 1782849742.82 total 119.06 last 198.35) - open/close/unlink 28920 (time 1782849752.82 total 129.07 last 236.17) - open/close/unlink 30000 (time 1782849757.84 total 134.09 last 215.10) - open/close/unlink 31981 (time 1782849767.84 total 144.09 last 198.08) - open/close/unlink 34328 (time 1782849777.84 total 154.09 last 234.68) - open/close/unlink 36756 (time 1782849787.84 total 164.09 last 242.77) - open/close/unlink 39064 (time 1782849797.85 total 174.10 last 230.69) - open/close/unlink 40000 (time 1782849802.01 total 178.26 last 224.79) - open/close/unlink 42313 (time 1782849812.01 total 188.26 last 231.26) - open/close/unlink 44728 (time 1782849822.02 total 198.26 last 241.43) - open/close/unlink 47056 (time 1782849832.02 total 208.27 last 232.77) - open/close/unlink 49390 (time 1782849842.02 total 218.27 last 233.37) - open/close/unlink 50000 (time 1782849844.68 total 220.93 last 228.80) - open/close/unlink 52353 (time 1782849854.68 total 230.93 last 235.29) - open/close/unlink 54688 (time 1782849864.69 total 240.93 last 233.47) - open/close/unlink 57062 (time 1782849874.69 total 250.94 last 237.33) - open/close/unlink 59446 (time 1782849884.69 total 260.94 last 238.36) - open/close/unlink 60000 (time 1782849887.02 total 263.26 last 238.25) - open/close/unlink 62382 (time 1782849897.02 total 273.27 last 238.14) - open/close/unlink 64602 (time 1782849907.02 total 283.27 last 222.00) total: 64768 open/close/unlink in 284.09 seconds: 227.98 ops/second - open/close/unlink 2322 (time 1782849633.75 total 10.00 last 232.11) - open/close/unlink 4420 (time 1782849643.76 total 20.01 last 209.68) - open/close/unlink 6617 (time 1782849653.76 total 30.01 last 219.63) - open/close/unlink 8760 (time 1782849663.76 total 40.01 last 214.24) - open/close/unlink 10000 (time 1782849668.93 total 45.19 last 239.84) - open/close/unlink 12211 (time 1782849678.94 total 55.19 last 221.05) - open/close/unlink 14458 (time 1782849688.94 total 65.19 last 224.62) - open/close/unlink 16786 (time 1782849698.94 total 75.19 last 232.74) - open/close/unlink 19092 (time 1782849708.94 total 85.19 last 230.59) - open/close/unlink 20000 (time 1782849712.70 total 88.95 last 241.91) - open/close/unlink 22321 (time 1782849722.70 total 98.95 last 232.08) - open/close/unlink 24591 (time 1782849732.70 total 108.95 last 226.98) - open/close/unlink 26505 (time 1782849742.70 total 118.95 last 191.32) - open/close/unlink 28858 (time 1782849752.70 total 128.95 last 235.29) - open/close/unlink 30000 (time 1782849757.93 total 134.18 last 218.54) - open/close/unlink 31955 (time 1782849767.93 total 144.18 last 195.48) - open/close/unlink 34293 (time 1782849777.93 total 154.18 last 233.73) - open/close/unlink 36682 (time 1782849787.93 total 164.18 last 238.87) - open/close/unlink 38968 (time 1782849797.93 total 174.19 last 228.55) - open/close/unlink 40000 (time 1782849802.67 total 178.93 last 217.80) - open/close/unlink 42360 (time 1782849812.67 total 188.93 last 235.98) - open/close/unlink 44752 (time 1782849822.68 total 198.93 last 239.15) - open/close/unlink 47103 (time 1782849832.68 total 208.93 last 234.97) - open/close/unlink 49420 (time 1782849842.68 total 218.94 last 231.65) - open/close/unlink 50000 (time 1782849845.17 total 221.42 last 233.67) - open/close/unlink 52353 (time 1782849855.17 total 231.42 last 235.30) - open/close/unlink 54700 (time 1782849865.17 total 241.42 last 234.68) - open/close/unlink 57083 (time 1782849875.17 total 251.42 last 238.29) - open/close/unlink 59455 (time 1782849885.20 total 261.46 last 236.31) - open/close/unlink 60000 (time 1782849887.51 total 263.76 last 236.69) - open/close/unlink 62400 (time 1782849897.55 total 273.80 last 239.09) - open/close/unlink 64650 (time 1782849907.55 total 283.81 last 224.81) total: 64768 open/close/unlink in 284.96 seconds: 227.29 ops/second - open/close/unlink 2373 (time 1782849633.65 total 10.01 last 237.18) - open/close/unlink 4520 (time 1782849643.65 total 20.01 last 214.65) - open/close/unlink 6738 (time 1782849653.65 total 30.01 last 221.78) - open/close/unlink 8886 (time 1782849663.66 total 40.01 last 214.74) - open/close/unlink 10000 (time 1782849668.34 total 44.70 last 237.59) - open/close/unlink 12214 (time 1782849678.35 total 54.71 last 221.28) - open/close/unlink 14520 (time 1782849688.35 total 64.71 last 230.55) - open/close/unlink 16833 (time 1782849698.36 total 74.71 last 231.19) - open/close/unlink 19083 (time 1782849708.36 total 84.71 last 224.96) - open/close/unlink 20000 (time 1782849712.13 total 88.49 last 243.07) - open/close/unlink 22337 (time 1782849722.14 total 98.49 last 233.61) - open/close/unlink 24578 (time 1782849732.14 total 108.49 last 224.02) - open/close/unlink 26502 (time 1782849742.14 total 118.50 last 192.30) - open/close/unlink 28813 (time 1782849752.15 total 128.50 last 231.05) - open/close/unlink 30000 (time 1782849757.56 total 133.92 last 219.22) - open/close/unlink 31975 (time 1782849767.56 total 143.92 last 197.44) - open/close/unlink 34254 (time 1782849777.57 total 153.92 last 227.87) - open/close/unlink 36614 (time 1782849787.57 total 163.92 last 235.93) - open/close/unlink 38909 (time 1782849797.57 total 173.92 last 229.48) - open/close/unlink 40000 (time 1782849802.60 total 178.95 last 216.99) - open/close/unlink 42380 (time 1782849812.60 total 188.96 last 237.93) - open/close/unlink 44782 (time 1782849822.60 total 198.96 last 240.15) - open/close/unlink 47160 (time 1782849832.60 total 208.96 last 237.73) - open/close/unlink 49474 (time 1782849842.61 total 218.96 last 231.32) - open/close/unlink 50000 (time 1782849844.92 total 221.28 last 227.53) - open/close/unlink 52343 (time 1782849854.92 total 231.28 last 234.29) - open/close/unlink 54696 (time 1782849864.92 total 241.28 last 235.25) - open/close/unlink 57053 (time 1782849874.92 total 251.28 last 235.68) - open/close/unlink 59423 (time 1782849884.93 total 261.28 last 236.95) - open/close/unlink 60000 (time 1782849887.36 total 263.71 last 237.27) - open/close/unlink 62370 (time 1782849897.36 total 273.72 last 236.93) - open/close/unlink 64600 (time 1782849907.36 total 283.72 last 222.98) total: 64768 open/close/unlink in 285.55 seconds: 226.82 ops/second - open/close/unlink 2350 (time 1782849633.21 total 10.00 last 234.98) - open/close/unlink 4470 (time 1782849643.21 total 20.00 last 211.94) - open/close/unlink 6691 (time 1782849653.21 total 30.00 last 222.09) - open/close/unlink 8794 (time 1782849663.22 total 40.01 last 210.25) - open/close/unlink 10000 (time 1782849668.35 total 45.13 last 235.18) - open/close/unlink 12217 (time 1782849678.35 total 55.14 last 221.59) - open/close/unlink 14468 (time 1782849688.35 total 65.14 last 225.02) - open/close/unlink 16786 (time 1782849698.35 total 75.14 last 231.80) - open/close/unlink 19071 (time 1782849708.36 total 85.15 last 228.44) - open/close/unlink 20000 (time 1782849712.19 total 88.98 last 242.25) - open/close/unlink 22329 (time 1782849722.19 total 98.98 last 232.88) - open/close/unlink 24560 (time 1782849732.19 total 108.98 last 223.09) - open/close/unlink 26488 (time 1782849742.19 total 118.98 last 192.76) - open/close/unlink 28777 (time 1782849752.20 total 128.99 last 228.85) - open/close/unlink 30000 (time 1782849757.82 total 134.61 last 217.40) - open/close/unlink 31955 (time 1782849767.82 total 144.61 last 195.49) - open/close/unlink 34287 (time 1782849777.82 total 154.61 last 233.17) - open/close/unlink 36624 (time 1782849787.83 total 164.62 last 233.65) - open/close/unlink 38908 (time 1782849797.83 total 174.62 last 228.40) - open/close/unlink 40000 (time 1782849802.82 total 179.61 last 218.74) - open/close/unlink 42312 (time 1782849812.82 total 189.61 last 231.15) - open/close/unlink 44683 (time 1782849822.82 total 199.61 last 237.09) - open/close/unlink 47005 (time 1782849832.82 total 209.61 last 232.15) - open/close/unlink 49323 (time 1782849842.83 total 219.62 last 231.72) - open/close/unlink 50000 (time 1782849845.78 total 222.57 last 229.21) - open/close/unlink 52330 (time 1782849855.78 total 232.57 last 232.94) - open/close/unlink 54671 (time 1782849865.79 total 242.58 last 234.02) - open/close/unlink 57030 (time 1782849875.79 total 252.58 last 235.80) - open/close/unlink 59401 (time 1782849885.79 total 262.58 last 237.06) - open/close/unlink 60000 (time 1782849888.26 total 265.05 last 242.75) - open/close/unlink 62368 (time 1782849898.26 total 275.05 last 236.79) - open/close/unlink 64462 (time 1782849908.27 total 285.06 last 209.20) total: 64768 open/close/unlink in 290.62 seconds: 222.87 ops/second - open/close/unlink 2327 (time 1782849633.64 total 10.00 last 232.62) - open/close/unlink 4428 (time 1782849643.64 total 20.01 last 210.03) - open/close/unlink 6636 (time 1782849653.64 total 30.01 last 220.79) - open/close/unlink 8752 (time 1782849663.64 total 40.01 last 211.56) - open/close/unlink 10000 (time 1782849668.86 total 45.23 last 239.14) - open/close/unlink 12203 (time 1782849678.87 total 55.23 last 220.19) - open/close/unlink 14446 (time 1782849688.87 total 65.24 last 224.18) - open/close/unlink 16778 (time 1782849698.88 total 75.24 last 233.12) - open/close/unlink 19086 (time 1782849708.88 total 85.24 last 230.76) - open/close/unlink 20000 (time 1782849712.63 total 89.00 last 243.30) - open/close/unlink 22352 (time 1782849722.64 total 99.00 last 235.14) - open/close/unlink 24592 (time 1782849732.64 total 109.00 last 223.96) - open/close/unlink 26512 (time 1782849742.64 total 119.00 last 192.00) - open/close/unlink 28839 (time 1782849752.64 total 129.01 last 232.66) - open/close/unlink 30000 (time 1782849757.99 total 134.36 last 217.01) - open/close/unlink 31944 (time 1782849767.99 total 144.36 last 194.37) - open/close/unlink 34257 (time 1782849777.99 total 154.36 last 231.26) - open/close/unlink 36618 (time 1782849787.99 total 164.36 last 236.09) - open/close/unlink 38900 (time 1782849798.00 total 174.36 last 228.13) - open/close/unlink 40000 (time 1782849802.94 total 179.30 last 222.69) - open/close/unlink 42326 (time 1782849812.94 total 189.31 last 232.51) - open/close/unlink 44717 (time 1782849822.94 total 199.31 last 239.02) - open/close/unlink 47004 (time 1782849832.95 total 209.31 last 228.69) - open/close/unlink 49334 (time 1782849842.95 total 219.31 last 232.95) - open/close/unlink 50000 (time 1782849845.94 total 222.30 last 222.60) - open/close/unlink 52336 (time 1782849855.94 total 232.30 last 233.60) - open/close/unlink 54693 (time 1782849865.94 total 242.31 last 235.64) - open/close/unlink 57047 (time 1782849875.94 total 252.31 last 235.39) - open/close/unlink 59421 (time 1782849885.95 total 262.31 last 237.32) - open/close/unlink 60000 (time 1782849888.33 total 264.70 last 242.60) - open/close/unlink 62366 (time 1782849898.33 total 274.70 last 236.55) - open/close/unlink 64412 (time 1782849908.34 total 284.70 last 204.58) total: 64768 open/close/unlink in 291.47 seconds: 222.21 ops/second Waiting for MDT destroys to complete oleg351-server: fstrim: /mnt/lustre-ost2: the discard operation is not supported pdsh@oleg351-client: oleg351-server: ssh exited with exit code 1 fail_loc=0 fail_val=0 umount lustre on /mnt/lustre..... Stopping client oleg351-client.virtnet /mnt/lustre (opts:) stop ost1 service on oleg351-server Stopping /mnt/lustre-ost1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds1 (opts:-f) on oleg351-server stop mds service on oleg351-server Stopping /mnt/lustre-mds2 (opts:-f) on oleg351-server unloading modules via unload_modules_local on: 'oleg351-server' oleg351-server: oleg351-server.virtnet: executing unload_modules_local oleg351-server: modules unloaded. pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-client: ssh exited with exit code 2 pdsh@oleg351-client: oleg351-server: ssh exited with exit code 2