[ 2910.740414] LDISKFS-fs (dm-0): recovery complete
[ 2910.743402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2911.000407] LustreError: 112496:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 2911.006232] LustreError: 112496:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2911.007034] Lustre: 112529:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2911.026638] Lustre: 112529:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2911.032195] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2911.071654] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1601
[ 2911.071675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1569
[ 2914.171108] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2916.324331] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2930.504925] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:41:18 (1761313278)
[ 2948.993813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2950.276325] Lustre: Failing over lustre-MDT0000
[ 2950.277956] Lustre: Skipped 13 previous similar messages
[ 2950.413987] Lustre: server umount lustre-MDT0000 complete
[ 2950.415400] Lustre: Skipped 13 previous similar messages
[ 2969.416655] LDISKFS-fs (dm-0): recovery complete
[ 2969.419910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2975.716026] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283ad9d645 to 0x27c83d283ada7ebc
[ 2975.726903] Lustre: Skipped 13 previous similar messages
[ 2976.255232] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2976.269375] Lustre: Skipped 8 previous similar messages
[ 2979.876278] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2981.445636] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 2981.449325] Lustre: Skipped 8 previous similar messages
[ 2981.487834] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1970 to 0x0:1985
[ 2981.491531] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2002 to 0x0:2017
[ 2987.825411] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2989.285449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3005.734486] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:42:33 (1761313353)
[ 3022.481914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3029.036831] LustreError: 116581:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000007:0x1:0x0].0xce09e17a (000000004f3204c6) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3039.713470] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313381/real 1761313381]  req@000000008913784f x1846867840131072/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313388 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0'
[ 3039.727934] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3039.732512] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3039.738432] LustreError: Skipped 13 previous similar messages
[ 3049.057748] LDISKFS-fs (dm-0): recovery complete
[ 3049.060398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3060.793978] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3065.667918] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2433
[ 3065.670321] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2386 to 0x0:2401
[ 3070.464927] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3072.016131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3088.260898] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 09:43:55 (1761313435)
[ 3089.545243] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3091.093459] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:43:58 (1761313438)
[ 3092.659519] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3093.326740] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3093.331635] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3093.339813] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2465
[ 3097.422429] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 09:44:05 (1761313445)
[ 3114.205202] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3142.175579] LDISKFS-fs (dm-2): recovery complete
[ 3142.181648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3144.389964] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2849
[ 3144.399250] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 3145.659366] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3198.459430] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:45:45 (1761313545)
[ 3205.791309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3209.185611] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3209.199898] LustreError: Skipped 269 previous similar messages
[ 3229.136891] LDISKFS-fs (dm-0): recovery complete
[ 3229.145380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3236.011796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3238.449398] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3238.451768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2881
[ 3244.021764] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3245.483678] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3245.536177] LustreError: 122270:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3245.541041] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3246.565124] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2881
[ 3262.528939] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 09:46:50 (1761313610)
[ 3266.857101] LustreError: 9676:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3272.160591] LustreError: 9676:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3272.170830] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3272.241433] LustreError: 41046:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3274.032241] LustreError: 7408:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3279.328083] LustreError: 7408:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3279.334441] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3279.345628] LustreError: 7408:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3279.349567] LustreError: 7408:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3280.595421] LustreError: 7408:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3285.985264] LustreError: 7408:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3285.995365] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3286.006919] Lustre: Skipped 1 previous similar message
[ 3287.443524] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3292.640365] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3294.023965] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3299.296117] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3299.299692] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3299.319603] Lustre: Skipped 1 previous similar message
[ 3299.330037] LustreError: 17875:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3307.361352] LustreError: 6236:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3307.364200] LustreError: 6236:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3312.609719] LustreError: 6236:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3312.614165] LustreError: 6236:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3319.264173] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3319.271457] Lustre: Skipped 3 previous similar messages
[ 3319.279872] LustreError: 6237:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3326.990888] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3326.995826] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3332.065791] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3332.069896] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3336.926278] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 09:48:04 (1761313684)
[ 3338.079631] LustreError: 6235:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3343.360798] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3344.327926] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3345.413436] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3347.558651] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3347.562033] Lustre: Skipped 1 previous similar message
[ 3352.047510] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3352.058695] Lustre: Skipped 4 previous similar messages
[ 3354.464075] LustreError: 6235:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3357.107956] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 09:48:25 (1761313705)
[ 3358.718045] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3358.732549] Lustre: Skipped 4 previous similar messages
[ 3362.666300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3374.014937] LDISKFS-fs (dm-0): recovery complete
[ 3374.018745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3374.190586] Lustre: *** cfs_fail_loc=712, val=0***
[ 3374.193549] LustreError: 44473:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000fd73024e x1846867840340032/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3374.204901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3374.248919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3374.251099] Lustre: Skipped 9 previous similar messages
[ 3374.291964] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3374.292157] LustreError: 126944:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 3374.295313] Lustre: Skipped 21 previous similar messages
[ 3374.300315] LustreError: 126944:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3374.308762] Lustre: 126978:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3374.313404] Lustre: 126978:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3374.317049] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3374.338791] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2913
[ 3374.350850] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2913
[ 3376.751227] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3379.682570] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3405.240142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3408.355908] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 3408.359719] Lustre: Skipped 44 previous similar messages
[ 3410.749733] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3413.515928] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2945
[ 3413.518375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2945
[ 3417.445937] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3418.609549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3425.651116] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 09:49:33 (1761313773)
[ 3425.786073] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3431.416504] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 09:49:39 (1761313779)
[ 3432.059488] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3432.061952] LustreError: 6245:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000003cdc60a0 x1846867831356992/t0(0) o700->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:711/0 lens 264/248 e 0 to 0 dl 1761313786 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3444.193469] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3444.205030] LustreError: Skipped 9 previous similar messages
[ 3458.249992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3467.776241] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up)
[ 3471.176905] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3473.423424] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3473.424116] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3478.544248] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3479.720262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3485.612675] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:50:33 (1761313833)
[ 3487.714119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3487.724424] Lustre: Skipped 33 previous similar messages
[ 3503.463650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3505.688390] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 3505.689909] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2988 to 0x0:3009
[ 3506.710680] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3514.369616] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3515.830329] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3585.497715] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:52:13 (1761313933)
[ 3590.865339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3592.258120] Lustre: Failing over lustre-MDT0000
[ 3592.259912] Lustre: Skipped 7 previous similar messages
[ 3592.377103] Lustre: server umount lustre-MDT0000 complete
[ 3592.378415] Lustre: Skipped 7 previous similar messages
[ 3612.109161] LDISKFS-fs (dm-0): recovery complete
[ 3612.112696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3620.331436] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283add021b to 0x27c83d283add12bb
[ 3620.347326] Lustre: Skipped 5 previous similar messages
[ 3622.043156] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3622.045748] Lustre: Skipped 6 previous similar messages
[ 3623.842442] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3626.213934] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 3626.218851] Lustre: Skipped 6 previous similar messages
[ 3626.238454] Lustre: *** cfs_fail_loc=216, val=0***
[ 3626.238883] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2998 to 0x0:3041
[ 3626.241299] LustreError: 133406:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 3627.299334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3020 to 0x0:3041
[ 3692.335452] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 09:54:00 (1761314040)
[ 3693.699322] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3693.702239] Lustre: Skipped 2 previous similar messages
[ 3693.714722] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3073
[ 3694.318852] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3105
[ 3703.244885] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 09:54:11 (1761314051)
[ 3715.041342] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314056/real 1761314056]  req@00000000a39727f4 x1846867840438592/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314063 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 3715.055719] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3715.061607] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3715.067577] LustreError: Skipped 5 previous similar messages
[ 3721.264255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3734.509599] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3736.556049] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 3736.557976] LustreError: 135091:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000038314c4a x1846867831411136/t0(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:292/0 lens 328/344 e 0 to 0 dl 1761314122 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 3775.489067] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:56
[ 3775.541575] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3073
[ 3775.541743] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3137
[ 3780.220235] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3781.237714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3788.148522] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 09:55:35 (1761314135)
[ 3789.940627] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3795.179740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3809.214742] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3809.233110] LustreError: Skipped 210 previous similar messages
[ 3815.507967] LDISKFS-fs (dm-0): recovery complete
[ 3815.510997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3824.944492] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3827.207712] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3105
[ 3827.211153] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 3832.252529] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3833.553721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3841.433949] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 09:56:28 (1761314188)
[ 3842.468397] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3851.308756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3852.769872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3872.340692] LDISKFS-fs (dm-0): recovery complete
[ 3872.347890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3885.757965] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3887.130394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3201
[ 3887.130515] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 3893.287554] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3894.656915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3902.117196] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 09:57:29 (1761314249)
[ 3903.016989] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3910.784994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3932.682920] LDISKFS-fs (dm-0): recovery complete
[ 3932.685979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3945.233449] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3947.550080] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 3947.553455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3233
[ 3954.183049] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 09:58:21 (1761314301)
[ 3956.041473] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3956.050365] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 3956.058819] LustreError: 6238:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000f13b3f8e x1846867831441152/t261993005072(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:480/0 lens 392/456 e 0 to 0 dl 1761314310 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 3975.421324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3976.330072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3976.338691] Lustre: Skipped 8 previous similar messages
[ 3976.372976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3976.378701] Lustre: Skipped 10 previous similar messages
[ 3979.272941] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3981.861489] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000b2b8cdea x1846867831441152/t261993005072(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:506/0 lens 392/456 e 0 to 0 dl 1761314336 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 3981.872801] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3265
[ 3981.872865] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 3986.650894] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3987.587340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3993.574494] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 09:59:01 (1761314341)
[ 3994.317252] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3994.329139] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000001cc968de x1846867831448768/t266287972368(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:553/0 lens 504/448 e 0 to 0 dl 1761314383 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 3999.681378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4020.422888] LDISKFS-fs (dm-0): recovery complete
[ 4020.428176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4025.827787] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 4025.834126] Lustre: Skipped 43 previous similar messages
[ 4028.745424] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4031.500809] Lustre: 9676:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000782f5bd1 x1846867831448768/t266287972368(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:591/0 lens 504/448 e 0 to 0 dl 1761314421 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4031.514974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3297
[ 4031.515076] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4037.463324] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4039.281288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4045.821750] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 09:59:53 (1761314393)
[ 4046.692623] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4046.694792] LustreError: 6236:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000001f153daa x1846867831457344/t270582939664(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:606/0 lens 504/448 e 0 to 0 dl 1761314436 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4048.504719] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4054.276322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4055.561840] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 4055.572795] Lustre: Skipped 2 previous similar messages
[ 4055.590459] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000024adb146 x1846867831457792/t270582939665(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:580/0 lens 392/456 e 0 to 0 dl 1761314410 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4055.613796] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4075.211834] LDISKFS-fs (dm-0): recovery complete
[ 4075.217576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4083.472973] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4086.310428] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4086.313767] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4092.292433] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:00:40 (1761314440)
[ 4093.091735] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4093.094690] Lustre: Skipped 1 previous similar message
[ 4093.098168] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000ae04beb2 x1846867831464832/t274877906960(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:652/0 lens 504/448 e 0 to 0 dl 1761314482 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4093.112987] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 4094.677819] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4094.680025] Lustre: Skipped 1 previous similar message
[ 4099.937539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4101.601418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4101.604136] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4101.615329] Lustre: Skipped 37 previous similar messages
[ 4101.618992] LustreError: Skipped 6 previous similar messages
[ 4119.794832] LDISKFS-fs (dm-0): recovery complete
[ 4119.797893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4123.668871] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4125.702026] Lustre: 6237:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000005b0c4775 x1846867831464832/t274877906960(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:685/0 lens 504/448 e 0 to 0 dl 1761314515 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4125.734305] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 4125.734569] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 4132.978973] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:01:20 (1761314480)
[ 4133.825289] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4135.535165] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4135.537092] LustreError: 43005:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000024adb146 x1846867831472128/t279172874256(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:660/0 lens 392/456 e 0 to 0 dl 1761314490 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4141.965254] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4142.596523] Lustre: 43005:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ba209d72 x1846867831472128/t279172874256(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:667/0 lens 392/456 e 0 to 0 dl 1761314497 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4162.722074] LDISKFS-fs (dm-0): recovery complete
[ 4162.729194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4173.223767] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4175.408297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 4175.427374] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4182.799491] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:02:10 (1761314530)
[ 4183.471137] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4183.478303] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4183.486214] Lustre: Skipped 1 previous similar message
[ 4183.493402] LustreError: 9676:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000072608f9e x1846867831478400/t283467841549(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:743/0 lens 664/600 e 0 to 0 dl 1761314573 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4227.099085] Lustre: 7408:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000002fa5bb7b x1846867831478400/t283467841549(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:31/0 lens 664/3424 e 0 to 0 dl 1761314616 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4232.827884] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:03:00 (1761314580)
[ 4238.943885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4240.282968] Lustre: Failing over lustre-MDT0000
[ 4240.285691] Lustre: Skipped 9 previous similar messages
[ 4240.440977] Lustre: server umount lustre-MDT0000 complete
[ 4240.442831] Lustre: Skipped 9 previous similar messages
[ 4259.621822] LDISKFS-fs (dm-0): recovery complete
[ 4259.624891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4265.454244] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283add5805 to 0x27c83d283add5e64
[ 4265.468730] Lustre: Skipped 9 previous similar messages
[ 4265.985834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4265.990795] Lustre: Skipped 9 previous similar messages
[ 4268.987205] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4271.126790] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted.
[ 4271.130276] Lustre: Skipped 9 previous similar messages
[ 4271.153973] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4271.154262] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 4276.245712] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4277.360304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4294.183680] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:04:01 (1761314641)
[ 4300.678657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4320.855235] LDISKFS-fs (dm-0): recovery complete
[ 4320.860679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4333.963344] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4336.187630] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 4336.190138] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 4341.435743] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4342.766334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4347.020404] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 4356.188928] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:05:03 (1761314703)
[ 4391.901669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4404.704147] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314746/real 1761314746]  req@00000000da43ac89 x1846867840639168/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314753 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 4404.719365] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 4404.722732] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4404.731403] LustreError: Skipped 10 previous similar messages
[ 4409.825320] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4409.835396] LustreError: Skipped 357 previous similar messages
[ 4413.165322] LDISKFS-fs (dm-0): recovery complete
[ 4413.171386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4425.746439] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4427.983677] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4673
[ 4427.984299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4708 to 0x0:4737
[ 4432.531078] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4433.558188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4492.424606] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 10:07:19 (1761314839)
[ 4499.337538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4520.451630] LDISKFS-fs (dm-0): recovery complete
[ 4520.455743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4524.417914] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4526.187732] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4705
[ 4526.188438] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4739 to 0x0:4769
[ 4533.660396] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4534.968916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4542.254389] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 4543.561328] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 4548.268656] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:08:15 (1761314895)
[ 4555.448776] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 4598.800791] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 4598.807941] Lustre: Skipped 2 previous similar messages
[ 4600.247254] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4600.249354] LustreError: 9676:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000006aabc8ce x1846867832963840/t300647710728(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:403/0 lens 66040/440 e 0 to 0 dl 1761314988 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 4643.849292] Lustre: 7408:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a5886cef x1846867832963840/t300647710728(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:447/0 lens 66040/440 e 0 to 0 dl 1761315032 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 4649.415704] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 4650.437389] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:09:58 (1761314998)
[ 4659.020751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4680.884361] LDISKFS-fs (dm-0): recovery complete
[ 4680.886756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4687.847267] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 4687.853117] Lustre: Skipped 39 previous similar messages
[ 4688.026985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4688.031083] Lustre: Skipped 8 previous similar messages
[ 4688.064730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4688.067731] Lustre: Skipped 8 previous similar messages
[ 4691.567927] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4694.348528] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4871 to 0x0:4897
[ 4694.349444] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4806 to 0x0:4833
[ 4699.214411] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4700.519416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4707.644721] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:10:55 (1761315055)
[ 4725.087615] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4735.456687] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4735.460932] LustreError: Skipped 4 previous similar messages
[ 4735.463903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4735.471441] Lustre: Skipped 25 previous similar messages
[ 4755.639056] LDISKFS-fs (dm-2): recovery complete
[ 4755.641487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4758.975352] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4771.916253] LustreError: 166195:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery
[ 4771.921858] Lustre: 165676:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4771.926213] Lustre: 165676:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 4771.930160] Lustre: 165676:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted:  req@000000007e292552 x1846867841067584/t0(17179870574) o6->lustre-MDT0000-mdtlov_UUID@0@lo:563/0 lens 544/0 e 0 to 0 dl 1761315148 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'osp-syn-0-0.0'
[ 4771.944261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)
[ 4771.944545] Lustre: 165676:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6
[ 4771.946715] Lustre: Skipped 3 previous similar messages
[ 4788.755224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4791.972793] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4801.002260] LustreError: 3336:0:(client.c:3161:ptlrpc_replay_interpret()) @@@ status 0, old was -19  req@00000000abca0968 x1846867841067584/t17179870574(17179870574) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 1761315182 ref 2 fl Interpret:RQU/4/0 rc 0/0 job:'osp-syn-0-0.0'
[ 4801.436450] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 4801.438685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5313
[ 4804.678734] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4805.718811] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4842.463557] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:13:10 (1761315190)
[ 4844.231767] Lustre: Failing over lustre-MDT0000
[ 4844.233179] Lustre: Skipped 6 previous similar messages
[ 4846.376353] Lustre: server umount lustre-MDT0000 complete
[ 4846.378107] Lustre: Skipped 6 previous similar messages
[ 4863.642480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4870.122311] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283ae19e20 to 0x27c83d283ae2ace9
[ 4870.129079] Lustre: Skipped 4 previous similar messages
[ 4871.301939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4871.310205] Lustre: Skipped 6 previous similar messages
[ 4873.838577] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4875.765815] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 4875.771477] Lustre: Skipped 6 previous similar messages
[ 4875.796461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5345
[ 4875.797498] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5249
[ 4902.946638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4906.089447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4908.550678] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5377
[ 4908.551386] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5281
[ 4912.092359] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4913.185099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4919.550049] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:14:27 (1761315267)
[ 4948.684419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4950.055300] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161
[ 4950.065729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5409
[ 4952.043304] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4959.017149] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4960.249224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4967.788474] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:15:15 (1761315315)
[ 4976.330524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4976.430468] Lustre: *** cfs_fail_loc=605, val=0***
[ 4976.432504] LustreError: 172247:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000002996f689 failed: rc = -95
[ 4976.441936] LustreError: 172247:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 4976.450649] LustreError: 172247:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 4976.457244] LustreError: 172247:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 4976.464366] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 4976.474714] LustreError: 172247:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 4976.489852] LustreError: 172247:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 4982.463971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4991.204893] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4993.579350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5441
[ 4993.582602] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5283 to 0x0:5313
[ 4997.380460] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 10:15:44 (1761315344)
[ 5003.158506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5010.441436] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5010.449100] LustreError: Skipped 202 previous similar messages
[ 5016.032140] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315357/real 1761315357]  req@0000000023cec3a9 x1846867841199104/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315364 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 5016.047538] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5016.051696] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5016.058609] LustreError: Skipped 5 previous similar messages
[ 5024.830744] LDISKFS-fs (dm-0): recovery complete
[ 5024.834039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5033.984840] Lustre: *** cfs_fail_loc=707, val=0***
[ 5035.335260] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5078.043431] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54
[ 5078.501491] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5455 to 0x0:5473
[ 5078.501492] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5326 to 0x0:5345
[ 5083.899811] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5085.099619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5092.462714] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 10:17:20 (1761315440)
[ 5118.945419] LustreError: 8389:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5121.178553] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply