[ 1630.824950] LustreError: 9702:0:(mdt_handler.c:7433:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 1630.829670] LustreError: 9702:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1630.834009] Lustre: 9847:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1630.839723] Lustre: 9847:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1630.844910] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1630.884156] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1507 to 0x0:1569
[ 1630.884157] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1576 to 0x0:1697
[ 1632.175040] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1637.783468] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 08:26:29 (1721219189)
[ 1645.678033] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1661.323300] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1670.555321] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1970 to 0x0:1985
[ 1670.555333] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113
[ 1672.414885] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1672.938554] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1681.064135] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 08:27:13 (1721219233)
[ 1686.756379] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1706.521393] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1715.607471] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2386 to 0x0:2401
[ 1715.607518] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529
[ 1717.567530] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1718.132566] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1726.230023] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 08:27:58 (1721219278)
[ 1726.850479] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 1727.429328] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 08:27:59 (1721219279)
[ 1728.211506] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 1728.580702] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1728.585122] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 1728.592642] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2386 to 0x0:2433
[ 1730.735327] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 08:28:02 (1721219282)
[ 1736.307724] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 1739.605041] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1740.593156] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 1741.241003] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1741.248392] LustreError: Skipped 1 previous similar message
[ 1744.611002] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1749.618456] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 1749.626435] LustreError: Skipped 1 previous similar message
[ 1752.928263] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1753.427343] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977
[ 1797.999828] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 08:29:10 (1721219350)
[ 1799.399973] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1816.258347] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1817.741220] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 1817.741237] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2849
[ 1819.893931] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1820.450907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1824.740495] LustreError: 23553:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 1824.741047] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 1825.750544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009
[ 1833.885356] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 08:29:45 (1721219385)
[ 1835.666392] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1840.669341] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1840.673478] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1841.362960] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1846.366326] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1846.370581] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1847.079330] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1852.085313] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1852.089809] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1852.764413] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1857.767308] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1858.469011] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1863.473313] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1863.477487] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1863.482213] Lustre: Skipped 1 previous similar message
[ 1869.814540] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1869.821238] LustreError: 24705:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 1874.827302] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1874.832642] LustreError: 24705:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 1880.514297] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1880.516393] Lustre: Skipped 2 previous similar messages
[ 1886.832696] LustreError: 25290:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 1886.834468] LustreError: 25290:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 1891.836355] LustreError: 25290:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 1891.840852] LustreError: 25290:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 1894.685759] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 08:30:46 (1721219446)
[ 1895.379330] LustreError: 25290:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1901.379354] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1906.386268] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1911.394051] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1916.402073] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1921.410071] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1931.426072] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1931.430590] Lustre: Skipped 1 previous similar message
[ 1935.384345] LustreError: 25290:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake
[ 1935.389498] Lustre: 25290:0:(service.c:2333:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout  req@ffff88009b2c0e40 x1804827441426880/t0(0) o38->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:0/0 lens 520/416 e 0 to 0 dl 1721219469 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 1936.433966] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 1936.439364] Lustre: Skipped 3 previous similar messages
[ 1937.118429] LustreError: 24705:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 1947.649134] Lustre: lustre-MDT0000: Export ffff88012d441000 already connecting from 192.168.204.5@tcp
[ 1947.653353] Lustre: Skipped 7 previous similar messages
[ 1948.825255] LustreError: 24705:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 1950.354991] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 08:31:42 (1721219502)
[ 1954.176964] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 1958.158088] Lustre: MGC192.168.204.105@tcp: Connection restored to  (at 0@lo)
[ 1958.161967] Lustre: Skipped 48 previous similar messages
[ 1958.301808] Lustre: *** cfs_fail_loc=712, val=0***
[ 1958.304878] LustreError: 22048:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@ffff88013711b900 x1804827446847744/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 1958.316256] LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 1958.348434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 1958.355108] Lustre: Skipped 17 previous similar messages
[ 1958.381084] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 1958.381254] LustreError: 29801:0:(mdt_handler.c:7433:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 1958.381259] LustreError: 29801:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 1958.393235] Lustre: Skipped 26 previous similar messages
[ 1958.395585] Lustre: 29924:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 1958.400570] Lustre: 29924:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 1958.405169] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 1958.439065] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2881
[ 1958.439067] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041
[ 1959.651851] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1975.344253] Lustre: 2852:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1721219521/real 1721219521]  req@ffff88012b5b8980 x1804827446852096/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1721219528 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 1975.353734] Lustre: 2852:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 79 previous similar messages
[ 1982.798006] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 1992.213626] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073
[ 1992.213635] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2834 to 0x0:2913
[ 1994.184541] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 1994.757151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 1998.190255] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 08:32:30 (1721219550)
[ 2001.370350] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 08:32:33 (1721219553)
[ 2001.717716] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 2001.721599] LustreError: 500:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88012c1af200 x1804827441449152/t0(0) o700->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:6/0 lens 264/248 e 0 to 0 dl 1721219561 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 2008.740819] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 2008.748545] Lustre: Skipped 4 previous similar messages
[ 2018.568358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2018.579456] Lustre: Skipped 32 previous similar messages
[ 2026.103099] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2026.754004] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 2026.759045] Lustre: Skipped 10 previous similar messages
[ 2026.780660] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 2026.785706] Lustre: Skipped 10 previous similar messages
[ 2026.810164] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 2026.810171] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2915 to 0x0:2945
[ 2030.051945] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2030.638086] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2034.214082] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 08:33:06 (1721219586)
[ 2034.978252] Lustre: Failing over lustre-OST0000
[ 2034.980395] Lustre: Skipped 16 previous similar messages
[ 2034.987562] LustreError: 3353:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items
[ 2034.992629] LustreError: 3353:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 72 previous similar messages
[ 2035.008963] Lustre: server umount lustre-OST0000 complete
[ 2035.011925] Lustre: Skipped 16 previous similar messages
[ 2035.762748] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2035.772472] LustreError: Skipped 1 previous similar message
[ 2049.075854] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2049.663471] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 2052.986282] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2053.543624] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2118.682649] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 08:34:30 (1721219670)
[ 2119.750660] LustreError: 5679:0:(osd_handler.c:694:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 2119.757798] LustreError: 5679:0:(osd_handler.c:694:osd_ro()) Skipped 11 previous similar messages
[ 2120.106297] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2130.840428] LustreError: 166-1: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2130.846241] LustreError: Skipped 13 previous similar messages
[ 2136.849246] Lustre: Evicted from MGS (at 192.168.204.105@tcp) after server handle changed from 0xd79386bab335822a to 0xd79386bab335913b
[ 2136.854710] Lustre: Skipped 13 previous similar messages
[ 2138.387766] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2146.122713] Lustre: *** cfs_fail_loc=216, val=0***
[ 2146.122721] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2966 to 0x0:3009
[ 2146.129837] LustreError: 6769:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 2147.138631] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 2208.438854] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 08:36:00 (1721219760)
[ 2209.102424] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2209.107550] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201
[ 2209.419897] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233
[ 2216.393020] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 08:36:08 (1721219768)
[ 2236.708953] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2245.078879] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 2245.081561] LustreError: 9917:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88012e16d580 x1804827441496512/t0(0) o101->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:249/0 lens 328/344 e 0 to 0 dl 1721219804 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 2252.080154] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 2252.127026] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 2252.127037] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3020 to 0x0:3041
[ 2254.022075] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2254.479106] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2258.215954] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 08:36:50 (1721219810)
[ 2259.595496] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2261.289902] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2280.882413] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2283.648688] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 2283.648706] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3020 to 0x0:3073
[ 2285.632668] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2286.125854] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2289.696839] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 08:37:21 (1721219841)
[ 2290.112556] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2292.739082] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2310.087069] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2313.676742] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 2313.676760] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3329
[ 2315.007617] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2315.496705] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2318.520627] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 08:37:50 (1721219870)
[ 2318.936137] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2321.590159] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2339.276574] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2342.988832] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3361
[ 2342.988860] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 2347.601095] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 08:38:19 (1721219899)
[ 2349.012365] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2349.014702] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2349.017699] LustreError: 18099:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88013512c740 x1804827441519744/t261993005073(0) o35->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:353/0 lens 392/456 e 0 to 0 dl 1721219908 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 2368.505582] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2373.043843] Lustre: 20273:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88008ee6c280 x1804827441519744/t261993005073(0) o35->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:377/0 lens 392/456 e 0 to 0 dl 1721219932 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 2373.059456] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3363 to 0x0:3393
[ 2373.060136] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3169
[ 2374.975511] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2375.537268] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2379.116238] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 08:38:51 (1721219931)
[ 2379.510054] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2379.512616] LustreError: 21585:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88009b6a8000 x1804827441526016/t266287972368(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:384/0 lens 504/448 e 0 to 0 dl 1721219939 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 2381.942686] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2402.684514] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2403.546613] Lustre: 23094:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009b212600 x1804827441526016/t266287972368(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:408/0 lens 504/448 e 0 to 0 dl 1721219963 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 2403.562465] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3201
[ 2403.565530] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 2406.624381] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2407.188490] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2410.671756] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 08:39:22 (1721219962)
[ 2411.061685] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2411.064270] LustreError: 23094:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a67dd0c0 x1804827441532736/t270582939664(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:415/0 lens 504/448 e 0 to 0 dl 1721219970 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 2412.390366] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2413.736110] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2430.728088] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2435.101751] Lustre: 25683:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88012b8ce3c0 x1804827441532864/t270582939665(0) o35->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:439/0 lens 392/456 e 0 to 0 dl 1721219994 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 2435.113596] Lustre: 25683:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 2435.118934] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 2435.122986] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3457
[ 2439.611759] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 08:39:51 (1721219991)
[ 2440.007769] LustreError: 25680:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88012eace880 x1804827441539200/t274877906960(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:444/0 lens 504/448 e 0 to 0 dl 1721219999 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 2440.018331] LustreError: 25680:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 2441.321451] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 2441.323232] Lustre: Skipped 1 previous similar message
[ 2442.914168] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2460.033273] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2464.046536] Lustre: 28132:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff880137119300 x1804827441539200/t274877906960(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:468/0 lens 504/448 e 0 to 0 dl 1721220023 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 2464.059182] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3265
[ 2464.059190] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3489
[ 2468.387974] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 08:40:20 (1721220020)
[ 2468.809305] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 2470.143359] Lustre: *** cfs_fail_loc=13b, val=315***
[ 2470.147358] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 2470.152084] Lustre: Skipped 2 previous similar messages
[ 2472.519072] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2489.302311] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2492.840807] Lustre: 30574:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800a6d7d580 x1804827441545472/t279172874256(0) o35->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:497/0 lens 392/456 e 0 to 0 dl 1721220052 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 2492.856543] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3491 to 0x0:3521
[ 2492.857233] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3297
[ 2497.386099] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 08:40:49 (1721220049)
[ 2497.727536] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 2497.730947] LustreError: 30572:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff88012eacf6c0 x1804827441550208/t283467841550(0) o101->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:502/0 lens 664/600 e 0 to 0 dl 1721220057 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 2497.742737] LustreError: 30572:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 2504.726584] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnecting
[ 2504.731724] Lustre: Skipped 1 previous similar message
[ 2504.736342] Lustre: 31931:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff88009b6aed40 x1804827441550208/t283467841550(0) o101->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:509/0 lens 664/3424 e 0 to 0 dl 1721220064 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 2506.842457] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 08:40:58 (1721220058)
[ 2508.230314] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2522.779981] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3523 to 0x0:3553
[ 2522.780054] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3329
[ 2523.485703] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2527.485508] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2528.051455] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2541.622804] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 08:41:33 (1721220093)
[ 2543.391168] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2561.129516] Lustre: MGC192.168.204.105@tcp: Connection restored to  (at 0@lo)
[ 2561.131650] Lustre: Skipped 44 previous similar messages
[ 2561.315013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2561.318061] Lustre: Skipped 14 previous similar messages
[ 2561.341930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2561.344528] Lustre: Skipped 16 previous similar messages
[ 2562.694684] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2571.834712] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3235 to 0x0:3361
[ 2571.834745] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3555 to 0x0:3585
[ 2573.696066] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2574.264391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2576.636478] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 2581.962284] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 08:42:13 (1721220133)
[ 2595.199644] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2603.344339] Lustre: 2850:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1721220149/real 1721220149]  req@ffff88012b87b440 x1804827446972096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1721220156 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 2603.357748] Lustre: 2850:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 81 previous similar messages
[ 2615.924764] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2620.950681] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865
[ 2620.951353] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4612 to 0x0:4641
[ 2622.865882] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2623.404993] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2647.177839] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 08:43:19 (1721220199)
[ 2648.796779] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2649.373643] Lustre: Failing over lustre-MDT0000
[ 2649.375051] Lustre: Skipped 13 previous similar messages
[ 2649.417166] LustreError: 10016:0:(lprocfs_jobstats.c:137:job_stat_exit()) should not have any items
[ 2649.418971] LustreError: 10016:0:(lprocfs_jobstats.c:137:job_stat_exit()) Skipped 65 previous similar messages
[ 2649.476185] Lustre: server umount lustre-MDT0000 complete
[ 2649.477869] Lustre: Skipped 13 previous similar messages
[ 2657.584458] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 2657.592740] Lustre: Skipped 30 previous similar messages
[ 2665.142630] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2665.330168] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2665.336003] Lustre: Skipped 14 previous similar messages
[ 2665.375389] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 2665.379994] Lustre: Skipped 14 previous similar messages
[ 2665.401486] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4643 to 0x0:4673
[ 2665.401488] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4897
[ 2669.067071] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2669.630582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2673.118453] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 2673.654299] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 2675.847241] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 08:43:47 (1721220227)
[ 2681.490473] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 2689.213410] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 2689.216142] Lustre: Skipped 1 previous similar message
[ 2689.218774] LustreError: 10915:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@ffff8800a70bd050 x1804827443019776/t300647710728(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:693/0 lens 66040/440 e 0 to 0 dl 1721220248 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 2696.211149] Lustre: 10915:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@ffff8800a7730000 x1804827443019776/t300647710728(0) o36->93683a9b-097d-479d-b87e-01df14b94c45@192.168.204.5@tcp:700/0 lens 66040/440 e 0 to 0 dl 1721220255 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 2698.870436] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 2699.448448] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 08:44:11 (1721220251)
[ 2702.969993] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2720.303147] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2721.534437] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025
[ 2721.534471] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4775 to 0x0:4801
[ 2724.255731] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2724.825395] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2728.891277] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 08:44:40 (1721220280)
[ 2734.467228] LustreError: 17107:0:(osd_handler.c:694:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 2734.473037] LustreError: 17107:0:(osd_handler.c:694:osd_ro()) Skipped 12 previous similar messages
[ 2734.791560] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2739.000727] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2739.006236] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2739.014922] LustreError: Skipped 5 previous similar messages
[ 2741.278447] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2746.274123] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2746.284172] LustreError: Skipped 1 previous similar message
[ 2752.828071] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2753.458514] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441
[ 2766.308062] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2766.315857] LustreError: Skipped 2 previous similar messages
[ 2778.224555] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2778.780204] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473
[ 2782.146100] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2782.673690] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2816.660468] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 08:46:08 (1721220368)
[ 2828.960349] LustreError: 166-1: MGC192.168.204.105@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 2828.966293] LustreError: Skipped 14 previous similar messages
[ 2834.969102] Lustre: Evicted from MGS (at 192.168.204.105@tcp) after server handle changed from 0xd79386bab33a14e3 to 0xd79386bab33b19e9
[ 2834.974655] Lustre: Skipped 14 previous similar messages
[ 2836.376423] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2845.457415] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5217
[ 2845.457431] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505
[ 2866.675941] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2873.480733] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537
[ 2873.480736] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5202 to 0x0:5249
[ 2875.311877] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2875.863565] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2879.210790] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 08:47:11 (1721220431)
[ 2893.475984] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.204.5@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2893.485500] LustreError: Skipped 5 previous similar messages
[ 2893.528622] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 2904.194448] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569
[ 2904.316842] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2908.161895] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 2908.697402] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 2912.600356] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 08:47:44 (1721220464)
[ 2916.579502] Lustre: *** cfs_fail_loc=605, val=0***
[ 2916.582005] LustreError: 28907:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa05532c0 failed: rc = -95
[ 2916.587296] LustreError: 28907:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 2916.591191] LustreError: 28907:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 2916.595289] LustreError: 28907:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 2916.599460] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 2916.603578] LustreError: 28907:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 2916.620028] LustreError: 28907:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 2920.136298] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2920.559373] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5251 to 0x0:5281
[ 2920.559376] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601
[ 2923.053008] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 08:47:55 (1721220475)
[ 2924.450010] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2943.355508] Lustre: DEBUG MARKER: oleg405-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2943.539605] Lustre: *** cfs_fail_loc=707, val=0***
[ 2950.559902] Lustre: lustre-MDT0000: Client 93683a9b-097d-479d-b87e-01df14b94c45 (at 192.168.204.5@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 2950.756487] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5614 to 0x0:5633
[ 2950.756499] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5295 to 0x0:5313
[ 2952.710557] Lustre: DEBUG MARKER: oleg405-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2953.280391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2957.381313] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 08:48:29 (1721220509)
[ 2980.824438] LustreError: 16921:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 2981.865097] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply