[ 2910.740414] LDISKFS-fs (dm-0): recovery complete
[ 2910.743402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2911.000407] LustreError: 112496:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 2911.006232] LustreError: 112496:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2911.007034] Lustre: 112529:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2911.026638] Lustre: 112529:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2911.032195] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 2911.071654] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1479 to 0x0:1601
[ 2911.071675] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1507 to 0x0:1569
[ 2914.171108] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2916.324331] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 2930.504925] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:41:18 (1761313278)
[ 2948.993813] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2950.276325] Lustre: Failing over lustre-MDT0000
[ 2950.277956] Lustre: Skipped 13 previous similar messages
[ 2950.413987] Lustre: server umount lustre-MDT0000 complete
[ 2950.415400] Lustre: Skipped 13 previous similar messages
[ 2969.416655] LDISKFS-fs (dm-0): recovery complete
[ 2969.419910] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 2975.716026] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283ad9d645 to 0x27c83d283ada7ebc
[ 2975.726903] Lustre: Skipped 13 previous similar messages
[ 2976.255232] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 2976.269375] Lustre: Skipped 8 previous similar messages
[ 2979.876278] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2981.445636] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 2981.449325] Lustre: Skipped 8 previous similar messages
[ 2981.487834] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1970 to 0x0:1985
[ 2981.491531] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2002 to 0x0:2017
[ 2987.825411] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2989.285449] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3005.734486] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:42:33 (1761313353)
[ 3022.481914] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3029.036831] LustreError: 116581:0:(ldlm_resource.c:1127:ldlm_resource_complain()) mdt-lustre-MDT0000_UUID: namespace resource [0x200000007:0x1:0x0].0xce09e17a (000000004f3204c6) refcount nonzero (1) after lock cleanup; forcing cleanup.
[ 3039.713470] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761313381/real 1761313381]  req@000000008913784f x1846867840131072/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761313388 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:4.0'
[ 3039.727934] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 8 previous similar messages
[ 3039.732512] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3039.738432] LustreError: Skipped 13 previous similar messages
[ 3049.057748] LDISKFS-fs (dm-0): recovery complete
[ 3049.060398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3060.793978] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3065.667918] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2433
[ 3065.670321] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2386 to 0x0:2401
[ 3070.464927] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3072.016131] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3088.260898] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 09:43:55 (1761313435)
[ 3089.545243] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 3091.093459] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:43:58 (1761313438)
[ 3092.659519] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 3093.326740] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3093.331635] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3093.339813] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2418 to 0x0:2465
[ 3097.422429] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 09:44:05 (1761313445)
[ 3114.205202] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 3142.175579] LDISKFS-fs (dm-2): recovery complete
[ 3142.181648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3144.389964] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2849
[ 3144.399250] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:65
[ 3145.659366] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3198.459430] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:45:45 (1761313545)
[ 3205.791309] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3209.185611] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3209.199898] LustreError: Skipped 269 previous similar messages
[ 3229.136891] LDISKFS-fs (dm-0): recovery complete
[ 3229.145380] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3236.011796] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3238.449398] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 3238.451768] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2881
[ 3244.021764] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3245.483678] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3245.536177] LustreError: 122270:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 3245.541041] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3246.565124] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2881
[ 3262.528939] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 09:46:50 (1761313610)
[ 3266.857101] LustreError: 9676:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3272.160591] LustreError: 9676:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3272.170830] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3272.241433] LustreError: 41046:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3274.032241] LustreError: 7408:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3279.328083] LustreError: 7408:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3279.334441] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3279.345628] LustreError: 7408:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3279.349567] LustreError: 7408:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 3280.595421] LustreError: 7408:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3285.985264] LustreError: 7408:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3285.995365] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3286.006919] Lustre: Skipped 1 previous similar message
[ 3287.443524] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3292.640365] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3294.023965] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3299.296117] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3299.299692] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3299.319603] Lustre: Skipped 1 previous similar message
[ 3299.330037] LustreError: 17875:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3307.361352] LustreError: 6236:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3307.364200] LustreError: 6236:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3312.609719] LustreError: 6236:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3312.614165] LustreError: 6236:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3319.264173] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3319.271457] Lustre: Skipped 3 previous similar messages
[ 3319.279872] LustreError: 6237:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3326.990888] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3326.995826] LustreError: 6237:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3332.065791] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3332.069896] LustreError: 6237:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3336.926278] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 09:48:04 (1761313684)
[ 3338.079631] LustreError: 6235:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3343.360798] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3344.327926] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3345.413436] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3347.558651] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3347.562033] Lustre: Skipped 1 previous similar message
[ 3352.047510] Lustre: lustre-MDT0000: Export 000000009c1085ab already connecting from 192.168.203.35@tcp
[ 3352.058695] Lustre: Skipped 4 previous similar messages
[ 3354.464075] LustreError: 6235:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3357.107956] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 09:48:25 (1761313705)
[ 3358.718045] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3358.732549] Lustre: Skipped 4 previous similar messages
[ 3362.666300] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3374.014937] LDISKFS-fs (dm-0): recovery complete
[ 3374.018745] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3374.190586] Lustre: *** cfs_fail_loc=712, val=0***
[ 3374.193549] LustreError: 44473:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000fd73024e x1846867840340032/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3374.204901] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
[ 3374.248919] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3374.251099] Lustre: Skipped 9 previous similar messages
[ 3374.291964] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3374.292157] LustreError: 126944:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 3374.295313] Lustre: Skipped 21 previous similar messages
[ 3374.300315] LustreError: 126944:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3374.308762] Lustre: 126978:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3374.313404] Lustre: 126978:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3374.317049] Lustre: lustre-MDT0000: disconnecting 2 stale clients
[ 3374.338791] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2913
[ 3374.350850] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2913
[ 3376.751227] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3379.682570] LustreError: lustre-MDT0000-osp-MDT0001: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
[ 3405.240142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3408.355908] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 3408.359719] Lustre: Skipped 44 previous similar messages
[ 3410.749733] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3413.515928] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2866 to 0x0:2945
[ 3413.518375] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2803 to 0x0:2945
[ 3417.445937] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3418.609549] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3425.651116] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 09:49:33 (1761313773)
[ 3425.786073] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 3431.416504] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 09:49:39 (1761313779)
[ 3432.059488] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3432.061952] LustreError: 6245:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000003cdc60a0 x1846867831356992/t0(0) o700->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:711/0 lens 264/248 e 0 to 0 dl 1761313786 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3444.193469] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 3444.205030] LustreError: Skipped 9 previous similar messages
[ 3458.249992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3467.776241] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (not set up)
[ 3471.176905] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3473.423424] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3473.424116] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2947 to 0x0:2977
[ 3478.544248] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3479.720262] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3485.612675] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:50:33 (1761313833)
[ 3487.714119] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3487.724424] Lustre: Skipped 33 previous similar messages
[ 3503.463650] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 3505.688390] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:97
[ 3505.689909] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2988 to 0x0:3009
[ 3506.710680] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3514.369616] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3515.830329] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3585.497715] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 09:52:13 (1761313933)
[ 3590.865339] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3592.258120] Lustre: Failing over lustre-MDT0000
[ 3592.259912] Lustre: Skipped 7 previous similar messages
[ 3592.377103] Lustre: server umount lustre-MDT0000 complete
[ 3592.378415] Lustre: Skipped 7 previous similar messages
[ 3612.109161] LDISKFS-fs (dm-0): recovery complete
[ 3612.112696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3620.331436] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283add021b to 0x27c83d283add12bb
[ 3620.347326] Lustre: Skipped 5 previous similar messages
[ 3622.043156] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 3622.045748] Lustre: Skipped 6 previous similar messages
[ 3623.842442] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3626.213934] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 3626.218851] Lustre: Skipped 6 previous similar messages
[ 3626.238454] Lustre: *** cfs_fail_loc=216, val=0***
[ 3626.238883] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2998 to 0x0:3041
[ 3626.241299] LustreError: 133406:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 3627.299334] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3020 to 0x0:3041
[ 3692.335452] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 09:54:00 (1761314040)
[ 3693.699322] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3693.702239] Lustre: Skipped 2 previous similar messages
[ 3693.714722] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3073
[ 3694.318852] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3105
[ 3703.244885] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 09:54:11 (1761314051)
[ 3715.041342] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314056/real 1761314056]  req@00000000a39727f4 x1846867840438592/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314063 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 3715.055719] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 3715.061607] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3715.067577] LustreError: Skipped 5 previous similar messages
[ 3721.264255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3734.509599] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3736.556049] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 3736.557976] LustreError: 135091:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000038314c4a x1846867831411136/t0(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:292/0 lens 328/344 e 0 to 0 dl 1761314122 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 3775.489067] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:56
[ 3775.541575] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3073
[ 3775.541743] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3052 to 0x0:3137
[ 3780.220235] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3781.237714] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3788.148522] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 09:55:35 (1761314135)
[ 3789.940627] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3795.179740] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3809.214742] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 3809.233110] LustreError: Skipped 210 previous similar messages
[ 3815.507967] LDISKFS-fs (dm-0): recovery complete
[ 3815.510997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3824.944492] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3827.207712] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3053 to 0x0:3105
[ 3827.211153] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 3832.252529] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3833.553721] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3841.433949] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 09:56:28 (1761314188)
[ 3842.468397] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3851.308756] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3852.769872] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3872.340692] LDISKFS-fs (dm-0): recovery complete
[ 3872.347890] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3885.757965] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3887.130394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3201
[ 3887.130515] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3107 to 0x0:3137
[ 3893.287554] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3894.656915] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3902.117196] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 09:57:29 (1761314249)
[ 3903.016989] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3910.784994] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3932.682920] LDISKFS-fs (dm-0): recovery complete
[ 3932.685979] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3945.233449] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3947.550080] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3139 to 0x0:3169
[ 3947.553455] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3233
[ 3954.183049] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 09:58:21 (1761314301)
[ 3956.041473] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3956.050365] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 3956.058819] LustreError: 6238:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000f13b3f8e x1846867831441152/t261993005072(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:480/0 lens 392/456 e 0 to 0 dl 1761314310 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 3975.421324] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 3976.330072] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3976.338691] Lustre: Skipped 8 previous similar messages
[ 3976.372976] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3976.378701] Lustre: Skipped 10 previous similar messages
[ 3979.272941] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3981.861489] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000b2b8cdea x1846867831441152/t261993005072(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:506/0 lens 392/456 e 0 to 0 dl 1761314336 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 3981.872801] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3265
[ 3981.872865] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 3986.650894] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3987.587340] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3993.574494] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 09:59:01 (1761314341)
[ 3994.317252] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3994.329139] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000001cc968de x1846867831448768/t266287972368(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:553/0 lens 504/448 e 0 to 0 dl 1761314383 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 3999.681378] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4020.422888] LDISKFS-fs (dm-0): recovery complete
[ 4020.428176] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4025.827787] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 4025.834126] Lustre: Skipped 43 previous similar messages
[ 4028.745424] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4031.500809] Lustre: 9676:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000782f5bd1 x1846867831448768/t266287972368(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:591/0 lens 504/448 e 0 to 0 dl 1761314421 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4031.514974] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3139 to 0x0:3297
[ 4031.515076] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 4037.463324] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4039.281288] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4045.821750] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 09:59:53 (1761314393)
[ 4046.692623] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4046.694792] LustreError: 6236:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000001f153daa x1846867831457344/t270582939664(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:606/0 lens 504/448 e 0 to 0 dl 1761314436 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4048.504719] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4054.276322] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4055.561840] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 4055.572795] Lustre: Skipped 2 previous similar messages
[ 4055.590459] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000024adb146 x1846867831457792/t270582939665(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:580/0 lens 392/456 e 0 to 0 dl 1761314410 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4055.613796] Lustre: 6238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 4075.211834] LDISKFS-fs (dm-0): recovery complete
[ 4075.217576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4083.472973] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4086.310428] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4086.313767] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 4092.292433] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:00:40 (1761314440)
[ 4093.091735] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4093.094690] Lustre: Skipped 1 previous similar message
[ 4093.098168] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000ae04beb2 x1846867831464832/t274877906960(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:652/0 lens 504/448 e 0 to 0 dl 1761314482 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 4093.112987] LustreError: 6237:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 4094.677819] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 4094.680025] Lustre: Skipped 1 previous similar message
[ 4099.937539] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4101.601418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4101.604136] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 4101.615329] Lustre: Skipped 37 previous similar messages
[ 4101.618992] LustreError: Skipped 6 previous similar messages
[ 4119.794832] LDISKFS-fs (dm-0): recovery complete
[ 4119.797893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4123.668871] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4125.702026] Lustre: 6237:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000005b0c4775 x1846867831464832/t274877906960(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:685/0 lens 504/448 e 0 to 0 dl 1761314515 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 4125.734305] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 4125.734569] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 4132.978973] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:01:20 (1761314480)
[ 4133.825289] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 4135.535165] Lustre: *** cfs_fail_loc=13b, val=315***
[ 4135.537092] LustreError: 43005:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000024adb146 x1846867831472128/t279172874256(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:660/0 lens 392/456 e 0 to 0 dl 1761314490 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 4141.965254] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4142.596523] Lustre: 43005:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ba209d72 x1846867831472128/t279172874256(0) o35->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:667/0 lens 392/456 e 0 to 0 dl 1761314497 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 4162.722074] LDISKFS-fs (dm-0): recovery complete
[ 4162.729194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4173.223767] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4175.408297] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 4175.427374] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 4182.799491] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:02:10 (1761314530)
[ 4183.471137] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 4183.478303] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 4183.486214] Lustre: Skipped 1 previous similar message
[ 4183.493402] LustreError: 9676:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000072608f9e x1846867831478400/t283467841549(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:743/0 lens 664/600 e 0 to 0 dl 1761314573 ref 1 fl Interpret:/0/0 rc 301/0 job:'touch.0'
[ 4227.099085] Lustre: 7408:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000002fa5bb7b x1846867831478400/t283467841549(0) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:31/0 lens 664/3424 e 0 to 0 dl 1761314616 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 4232.827884] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:03:00 (1761314580)
[ 4238.943885] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4240.282968] Lustre: Failing over lustre-MDT0000
[ 4240.285691] Lustre: Skipped 9 previous similar messages
[ 4240.440977] Lustre: server umount lustre-MDT0000 complete
[ 4240.442831] Lustre: Skipped 9 previous similar messages
[ 4259.621822] LDISKFS-fs (dm-0): recovery complete
[ 4259.624891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4265.454244] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283add5805 to 0x27c83d283add5e64
[ 4265.468730] Lustre: Skipped 9 previous similar messages
[ 4265.985834] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4265.990795] Lustre: Skipped 9 previous similar messages
[ 4268.987205] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4271.126790] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted.
[ 4271.130276] Lustre: Skipped 9 previous similar messages
[ 4271.153973] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3395 to 0x0:3425
[ 4271.154262] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3361
[ 4276.245712] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4277.360304] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4294.183680] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:04:01 (1761314641)
[ 4300.678657] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4320.855235] LDISKFS-fs (dm-0): recovery complete
[ 4320.860679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4333.963344] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4336.187630] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 4336.190138] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3393
[ 4341.435743] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4342.766334] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4347.020404] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 4356.188928] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:05:03 (1761314703)
[ 4391.901669] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4404.704147] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314746/real 1761314746]  req@00000000da43ac89 x1846867840639168/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314753 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 4404.719365] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 10 previous similar messages
[ 4404.722732] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4404.731403] LustreError: Skipped 10 previous similar messages
[ 4409.825320] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4409.835396] LustreError: Skipped 357 previous similar messages
[ 4413.165322] LDISKFS-fs (dm-0): recovery complete
[ 4413.171386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4425.746439] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4427.983677] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4673
[ 4427.984299] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4708 to 0x0:4737
[ 4432.531078] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4433.558188] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4492.424606] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 10:07:19 (1761314839)
[ 4499.337538] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4520.451630] LDISKFS-fs (dm-0): recovery complete
[ 4520.455743] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4524.417914] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4526.187732] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4705
[ 4526.188438] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4739 to 0x0:4769
[ 4533.660396] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4534.968916] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4542.254389] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 4543.561328] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 4548.268656] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:08:15 (1761314895)
[ 4555.448776] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 4598.800791] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 4598.807941] Lustre: Skipped 2 previous similar messages
[ 4600.247254] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4600.249354] LustreError: 9676:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000006aabc8ce x1846867832963840/t300647710728(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:403/0 lens 66040/440 e 0 to 0 dl 1761314988 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 4643.849292] Lustre: 7408:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000a5886cef x1846867832963840/t300647710728(0) o36->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:447/0 lens 66040/440 e 0 to 0 dl 1761315032 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 4649.415704] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 4650.437389] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:09:58 (1761314998)
[ 4659.020751] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4680.884361] LDISKFS-fs (dm-0): recovery complete
[ 4680.886756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4687.847267] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 4687.853117] Lustre: Skipped 39 previous similar messages
[ 4688.026985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4688.031083] Lustre: Skipped 8 previous similar messages
[ 4688.064730] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4688.067731] Lustre: Skipped 8 previous similar messages
[ 4691.567927] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4694.348528] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4871 to 0x0:4897
[ 4694.349444] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4806 to 0x0:4833
[ 4699.214411] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4700.519416] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4707.644721] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:10:55 (1761315055)
[ 4725.087615] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4735.456687] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4735.460932] LustreError: Skipped 4 previous similar messages
[ 4735.463903] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4735.471441] Lustre: Skipped 25 previous similar messages
[ 4755.639056] LDISKFS-fs (dm-2): recovery complete
[ 4755.641487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4758.975352] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4771.916253] LustreError: 166195:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-OST0000: Aborting recovery
[ 4771.921858] Lustre: 165676:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 4771.926213] Lustre: 165676:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 4771.930160] Lustre: 165676:0:(ldlm_lib.c:1803:abort_req_replay_queue()) @@@ aborted:  req@000000007e292552 x1846867841067584/t0(17179870574) o6->lustre-MDT0000-mdtlov_UUID@0@lo:563/0 lens 544/0 e 0 to 0 dl 1761315148 ref 1 fl Complete:/4/ffffffff rc 0/-1 job:'osp-syn-0-0.0'
[ 4771.944261] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)
[ 4771.944545] Lustre: 165676:0:(ofd_obd.c:554:ofd_postrecov()) lustre-OST0000: auto trigger paused LFSCK failed: rc = -6
[ 4771.946715] Lustre: Skipped 3 previous similar messages
[ 4788.755224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4791.972793] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4801.002260] LustreError: 3336:0:(client.c:3161:ptlrpc_replay_interpret()) @@@ status 0, old was -19  req@00000000abca0968 x1846867841067584/t17179870574(17179870574) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 1761315182 ref 2 fl Interpret:RQU/4/0 rc 0/0 job:'osp-syn-0-0.0'
[ 4801.436450] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:129
[ 4801.438685] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5313
[ 4804.678734] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4805.718811] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4842.463557] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:13:10 (1761315190)
[ 4844.231767] Lustre: Failing over lustre-MDT0000
[ 4844.233179] Lustre: Skipped 6 previous similar messages
[ 4846.376353] Lustre: server umount lustre-MDT0000 complete
[ 4846.378107] Lustre: Skipped 6 previous similar messages
[ 4863.642480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4870.122311] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283ae19e20 to 0x27c83d283ae2ace9
[ 4870.129079] Lustre: Skipped 4 previous similar messages
[ 4871.301939] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4871.310205] Lustre: Skipped 6 previous similar messages
[ 4873.838577] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4875.765815] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 4875.771477] Lustre: Skipped 6 previous similar messages
[ 4875.796461] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5345
[ 4875.797498] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5249
[ 4902.946638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4906.089447] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4908.550678] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5298 to 0x0:5377
[ 4908.551386] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5281
[ 4912.092359] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4913.185099] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4919.550049] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:14:27 (1761315267)
[ 4948.684419] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 4950.055300] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:34 to 0x280000400:161
[ 4950.065729] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5409
[ 4952.043304] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4959.017149] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4960.249224] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4967.788474] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:15:15 (1761315315)
[ 4976.330524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4976.430468] Lustre: *** cfs_fail_loc=605, val=0***
[ 4976.432504] LustreError: 172247:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=000000002996f689 failed: rc = -95
[ 4976.441936] LustreError: 172247:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 4976.450649] LustreError: 172247:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 4976.457244] LustreError: 172247:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 4976.464366] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 4976.474714] LustreError: 172247:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 4976.489852] LustreError: 172247:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 4982.463971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 4991.204893] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4993.579350] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5379 to 0x0:5441
[ 4993.582602] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5283 to 0x0:5313
[ 4997.380460] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 10:15:44 (1761315344)
[ 5003.158506] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5010.441436] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5010.449100] LustreError: Skipped 202 previous similar messages
[ 5016.032140] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315357/real 1761315357]  req@0000000023cec3a9 x1846867841199104/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315364 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 5016.047538] Lustre: 3340:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 5 previous similar messages
[ 5016.051696] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5016.058609] LustreError: Skipped 5 previous similar messages
[ 5024.830744] LDISKFS-fs (dm-0): recovery complete
[ 5024.834039] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5033.984840] Lustre: *** cfs_fail_loc=707, val=0***
[ 5035.335260] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5078.043431] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:54
[ 5078.501491] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5455 to 0x0:5473
[ 5078.501492] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5326 to 0x0:5345
[ 5083.899811] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5085.099619] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5092.462714] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 10:17:20 (1761315440)
[ 5118.945419] LustreError: 8389:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 5121.178553] Lustre: DEBUG MARKER: replay-single test_65a: @@@@@@ FAIL: No early reply
[ 5124.016224] LustreError: 8389:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 5125.478240] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:17:53 (1761315473)
[ 5153.085596] LustreError: 8213:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 5159.136263] LustreError: 8213:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 5164.063639] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:18:31 (1761315511)
[ 5189.568653] LustreError: 6236:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 5194.577723] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5195.967035] LustreError: 6236:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 5206.064317] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5222.745349] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:19:30 (1761315570)
[ 5284.052665] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:20:31 (1761315631)
[ 5310.829927] LustreError: 7408:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5311.273666] LustreError: 7408:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5315.528132] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5315.534562] LustreError: 6236:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 15 previous similar messages
[ 5318.990297] LustreError: 43005:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5318.996171] LustreError: 43005:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 25 previous similar messages
[ 5323.696118] LustreError: 43005:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 5323.700116] LustreError: 43005:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 24 previous similar messages
[ 5335.010503] LustreError: 127397:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 5335.013510] LustreError: 127397:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 64 previous similar messages
[ 5343.345688] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:21:30 (1761315690)
[ 5374.222756] Lustre: DEBUG MARKER: phase 2
[ 5381.148797] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 10:22:08 (1761315728)
[ 5460.575964] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 10:23:28 (1761315808)
[ 5461.624908] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 5463.224807] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 2mdts recovery; 1 clients ========================================================== 10:23:30 (1761315810)
[ 5467.284327] Lustre: DEBUG MARKER: Started rundbench load pid=138495 ...
[ 5474.879881] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5477.348431] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 5478.869787] Lustre: Failing over lustre-MDT0000
[ 5478.877509] Lustre: Skipped 4 previous similar messages
[ 5479.128761] Lustre: server umount lustre-MDT0000 complete
[ 5479.134693] Lustre: Skipped 5 previous similar messages
[ 5480.423717] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 5480.430410] LustreError: Skipped 5 previous similar messages
[ 5480.435518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5480.446425] Lustre: Skipped 19 previous similar messages
[ 5499.590051] LDISKFS-fs (dm-0): recovery complete
[ 5499.593529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5507.558397] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283ae2bfc0 to 0x27c83d283ae3e27f
[ 5507.569991] Lustre: Skipped 3 previous similar messages
[ 5507.581487] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 5507.583596] Lustre: Skipped 28 previous similar messages
[ 5507.854181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5507.859121] Lustre: Skipped 7 previous similar messages
[ 5507.896350] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5507.901085] Lustre: Skipped 7 previous similar messages
[ 5508.096977] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 5508.102622] Lustre: Skipped 4 previous similar messages
[ 5511.377321] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5513.265609] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted.
[ 5513.269428] Lustre: Skipped 4 previous similar messages
[ 5513.312965] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5542 to 0x0:5569
[ 5513.313307] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5395 to 0x0:5441
[ 5519.171731] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5520.586918] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5529.264743] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 5531.813326] Lustre: DEBUG MARKER: test_70b fail mds2 2 times
[ 5533.701918] LustreError: 181725:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.35@tcp arrived at 1761315882 with bad export cookie 2866608405816563811
[ 5533.739818] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping)
[ 5533.758236] LustreError: 6235:0:(ldlm_lockd.c:1427:ldlm_handle_enqueue0()) ### lock on destroyed export 00000000cc31dbb4 ns: mdt-lustre-MDT0001_UUID lock: 00000000a92444ab/0x27c83d283ae46f58 lrc: 3/0,0 mode: PR/PR res: [0x240000403:0x162:0x0].0x0 bits 0x1b/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 192.168.203.35@tcp remote: 0x829bcdd5c72838d0 expref: 2 pid: 6235 timeout: 0 lvb_type: 0
[ 5554.733461] LDISKFS-fs (dm-1): recovery complete
[ 5554.738241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5557.951768] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5561.191442] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:332 to 0x280000400:353
[ 5561.193441] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:171 to 0x2c0000400:193
[ 5565.724442] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 5567.001884] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5574.913775] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5577.252431] Lustre: DEBUG MARKER: test_70b fail mds1 3 times
[ 5578.776711] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping)
[ 5598.694048] LDISKFS-fs (dm-0): recovery complete
[ 5598.697218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5608.818775] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5611.091983] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5395 to 0x0:5473
[ 5611.096205] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5542 to 0x0:5601
[ 5615.902743] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5617.180808] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5623.907146] Lustre: DEBUG MARKER: == replay-single test 70c: tar 2mdts recovery ============ 10:26:11 (1761315971)
[ 5751.041452] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 5762.544678] Lustre: DEBUG MARKER: test_70c fail mds2 1 times
[ 5764.038025] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping)
[ 5764.578158] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5764.586812] LustreError: Skipped 123 previous similar messages
[ 5784.109492] LDISKFS-fs (dm-1): recovery complete
[ 5784.115731] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5787.419890] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5792.616435] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:817 to 0x280000400:833
[ 5792.616711] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:656 to 0x2c0000400:673
[ 5796.313521] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 5797.390768] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5924.849223] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5936.008652] Lustre: DEBUG MARKER: test_70c fail mds1 2 times
[ 5936.959387] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping)
[ 5945.312120] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316286/real 1761316286]  req@0000000011d768bf x1846867842875840/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761316293 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 5945.320236] Lustre: 3338:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
[ 5945.322725] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5945.326996] LustreError: Skipped 2 previous similar messages
[ 5954.880585] LDISKFS-fs (dm-0): recovery complete
[ 5954.886201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 5965.223904] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5970.940341] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6206 to 0x0:6241
[ 5970.940703] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6333 to 0x0:6369
[ 5976.477667] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5977.793456] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6028.076299] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 2mdts recovery ========================================================== 10:32:55 (1761316375)
[ 6155.706898] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6167.623915] Lustre: DEBUG MARKER: test_70d fail mds1 1 times
[ 6169.093393] Lustre: Failing over lustre-MDT0000
[ 6169.095194] Lustre: Skipped 4 previous similar messages
[ 6169.176060] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping)
[ 6169.274699] Lustre: server umount lustre-MDT0000 complete
[ 6169.276521] Lustre: Skipped 4 previous similar messages
[ 6170.594786] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107
[ 6170.599106] LustreError: Skipped 3 previous similar messages
[ 6170.601069] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6170.610189] Lustre: Skipped 17 previous similar messages
[ 6190.138538] LDISKFS-fs (dm-0): recovery complete
[ 6190.140896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6195.685261] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283af06c8b to 0x27c83d283af6648a
[ 6195.698340] Lustre: Skipped 2 previous similar messages
[ 6195.705371] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 6195.709871] Lustre: Skipped 20 previous similar messages
[ 6195.933677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 6195.936910] Lustre: Skipped 4 previous similar messages
[ 6195.954552] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6195.958304] Lustre: Skipped 4 previous similar messages
[ 6196.223162] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 6196.231574] Lustre: Skipped 4 previous similar messages
[ 6199.548515] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6202.978172] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted.
[ 6202.991170] Lustre: Skipped 4 previous similar messages
[ 6203.035499] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6556 to 0x0:6593
[ 6203.039925] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6428 to 0x0:6465
[ 6207.387509] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6208.803257] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6215.146164] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:36:02 (1761316562)
[ 6340.758722] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 6352.122884] Lustre: DEBUG MARKER: test_70e fail mds2 1 times
[ 6353.480619] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 6365.156587] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6365.170898] LustreError: Skipped 93 previous similar messages
[ 6372.978229] LDISKFS-fs (dm-1): recovery complete
[ 6372.980682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6376.158436] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6378.509560] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1378 to 0x280000400:1409
[ 6378.510501] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1218 to 0x2c0000400:1249
[ 6383.039597] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 6384.225349] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6512.324856] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6523.706957] Lustre: DEBUG MARKER: test_70e fail mds1 2 times
[ 6524.999959] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.35@tcp (stopping)
[ 6544.596818] LDISKFS-fs (dm-0): recovery complete
[ 6544.603029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6553.983197] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6561.670719] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7897 to 0x0:7937
[ 6561.671866] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8025 to 0x0:8065
[ 6566.032170] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 6567.438574] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 6574.203722] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 10:42:01 (1761316921)
[ 6584.094852] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6586.285174] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 6606.908085] LDISKFS-fs (dm-2): recovery complete
[ 6606.910828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6608.918119] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1378 to 0x280000400:1441
[ 6608.918764] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8129
[ 6609.720343] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6616.748905] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6617.839632] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6631.166268] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 6633.533179] Lustre: DEBUG MARKER: test_70f failing OST 2 times
[ 6634.507250] Lustre: lustre-OST0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnecting
[ 6634.511687] Lustre: Skipped 1 previous similar message
[ 6659.114387] LDISKFS-fs (dm-2): recovery complete
[ 6659.117316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 6661.165499] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1378 to 0x280000400:1473
[ 6661.170762] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8161
[ 6664.199830] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6673.511271] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 6675.220169] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 6684.352343] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 10:43:51 (1761317031)
[ 6811.200071] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 6816.663315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 6828.276498] Lustre: DEBUG MARKER: fail mds2 mds1 1 times
[ 6829.573724] Lustre: Failing over lustre-MDT0001
[ 6829.575448] Lustre: Skipped 4 previous similar messages
[ 6829.592787] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.35@tcp (stopping)
[ 6829.596147] Lustre: Skipped 1 previous similar message
[ 6829.688495] Lustre: server umount lustre-MDT0001 complete
[ 6829.690507] Lustre: Skipped 4 previous similar messages
[ 6832.097800] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 6832.108031] LustreError: Skipped 3 previous similar messages
[ 6832.110290] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 6832.126617] Lustre: Skipped 14 previous similar messages
[ 6840.800630] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317182/real 1761317182]  req@000000008357b066 x1846867846447168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317189 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 6840.801640] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 6840.817525] Lustre: 3339:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 4 previous similar messages
[ 6840.823185] LustreError: Skipped 2 previous similar messages
[ 6853.300052] LDISKFS-fs (dm-1): recovery complete
[ 6853.302574] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6955.500607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 6955.504871] Lustre: Skipped 4 previous similar messages
[ 6955.526348] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 6955.529299] Lustre: Skipped 4 previous similar messages
[ 6956.030921] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 6956.035678] Lustre: Skipped 4 previous similar messages
[ 6958.462848] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6960.620650] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to  (at 0@lo)
[ 6960.622977] Lustre: Skipped 16 previous similar messages
[ 6965.730325] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6965.754803] LustreError: Skipped 177 previous similar messages
[ 6978.942304] LDISKFS-fs (dm-0): recovery complete
[ 6978.961728] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 6994.414909] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283b014900 to 0x27c83d283b0696dd
[ 6994.419510] Lustre: Skipped 1 previous similar message
[ 6998.909144] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7000.639344] Lustre: lustre-MDT0001: Recovery over after 0:44, of 2 clients 2 recovered and 0 were evicted.
[ 7000.642358] Lustre: Skipped 4 previous similar messages
[ 7000.707577] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1378 to 0x280000400:1505
[ 7000.708296] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1218 to 0x2c0000400:1281
[ 7008.113144] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8193
[ 7008.117837] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7983 to 0x0:8001
[ 7013.571595] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7014.731892] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7015.994551] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7023.198167] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 10:49:30 (1761317370)
[ 7029.716236] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7053.157147] LDISKFS-fs (dm-0): recovery complete
[ 7053.167128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7060.425686] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 7063.387776] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7067.669372] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 7067.710048] Lustre: 203119:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000000ad762e4 x1846867845489728/t343597384384(343597384384) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:572/0 lens 648/3424 e 0 to 0 dl 1761317422 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0'
[ 7067.784662] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8003 to 0x0:8033
[ 7067.786269] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8225
[ 7071.893703] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7073.019233] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7079.372799] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 10:50:26 (1761317426)
[ 7084.784557] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7107.652837] LDISKFS-fs (dm-0): recovery complete
[ 7107.655443] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7116.377399] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 7116.384708] LustreError: 203103:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000064d8409 x1846867845489728/t343597384384(343597384384) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:620/0 lens 648/600 e 0 to 0 dl 1761317470 ref 1 fl Interpret:/4/0 rc 301/0 job:'lfs.0'
[ 7119.068720] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7123.476535] Lustre: lustre-MDT0000: Client e2b61cea-b3fc-4b27-8653-b4f72334e01b (at 192.168.203.35@tcp) reconnected, waiting for 2 clients in recovery for 0:58
[ 7123.496774] Lustre: 203561:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000ba6893f5 x1846867845489728/t343597384384(343597384384) o101->e2b61cea-b3fc-4b27-8653-b4f72334e01b@192.168.203.35@tcp:628/0 lens 648/3424 e 0 to 0 dl 1761317478 ref 1 fl Interpret:/6/0 rc 0/0 job:'lfs.0'
[ 7123.601550] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8065
[ 7123.605120] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8257
[ 7127.382348] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7128.621080] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7135.001494] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 10:51:22 (1761317482)
[ 7157.501512] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7168.013432] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7169.528726] Lustre: lustre-MDT0000: Denying connection for new client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 7169.539796] Lustre: Skipped 12 previous similar messages
[ 7170.574463] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8097
[ 7178.972491] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 7179.777646] Lustre: lustre-OST0000: Denying connection for new client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp), waiting for 2 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 7181.212734] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1378 to 0x280000400:1537
[ 7181.216049] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8111 to 0x0:8289
[ 7182.395733] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7191.601792] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 10:52:19 (1761317539)
[ 7192.375160] Lustre: *** cfs_fail_loc=1701, val=2147483648***
[ 7192.380231] LustreError: 203106:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000f7b4ee82 x1846867846702912/t360777252871(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:696/0 lens 1056/4320 e 0 to 0 dl 1761317546 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 7198.499166] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7198.693658] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7218.532856] LDISKFS-fs (dm-0): recovery complete
[ 7218.535549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7231.173701] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7234.105361] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8129
[ 7234.106711] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8321
[ 7238.346164] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7239.478072] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7245.885978] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 10:53:13 (1761317593)
[ 7246.753864] LustreError: 206292:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000e7acb61d x1846867846723136/t365072220170(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:751/0 lens 2200/4320 e 0 to 0 dl 1761317601 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 7252.298236] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7253.986484] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7259.032837] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7281.956226] LDISKFS-fs (dm-1): recovery complete
[ 7281.959062] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7285.610444] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7287.343486] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1292 to 0x2c0000400:1313
[ 7287.346600] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1548 to 0x280000400:1569
[ 7293.957404] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7295.359918] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7303.116650] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 10:54:10 (1761317650)
[ 7310.140164] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7311.330330] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7315.694768] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7337.074691] LDISKFS-fs (dm-0): recovery complete
[ 7337.076969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7346.155211] LustreError: 217511:0:(ldlm_resource.c:1127:ldlm_resource_complain()) MGC192.168.203.135@tcp: namespace resource [0x65727473756c:0x5:0x0].0x0 (00000000f989152c) refcount nonzero (2) after lock cleanup; forcing cleanup.
[ 7346.171443] LustreError: 217511:0:(ldlm_resource.c:1127:ldlm_resource_complain()) Skipped 7 previous similar messages
[ 7348.879493] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7351.939798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8353
[ 7351.944539] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8161
[ 7356.068995] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7357.295097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7379.331217] LDISKFS-fs (dm-1): recovery complete
[ 7379.334470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7382.251120] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7384.591255] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1580 to 0x280000400:1601
[ 7384.591453] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1324 to 0x2c0000400:1345
[ 7389.556323] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7390.833367] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7398.225814] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 10:55:45 (1761317745)
[ 7399.028627] Lustre: *** cfs_fail_loc=1701, val=2147483648***
[ 7399.030040] Lustre: Skipped 2 previous similar messages
[ 7399.031286] LustreError: 203106:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000097075443 x1846867846776256/t369367187470(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:148/0 lens 2200/4320 e 0 to 0 dl 1761317753 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 7399.038228] LustreError: 203106:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 7405.539493] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7407.633837] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7414.044957] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7415.779484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 7419.062153] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317767 with bad export cookie 2866608405819636608
[ 7419.070884] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 1 previous similar message
[ 7440.753459] LDISKFS-fs (dm-0): recovery complete
[ 7440.757335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7454.759840] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7475.132918] LDISKFS-fs (dm-1): recovery complete
[ 7475.138102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7475.309528] LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_connect to node 0@lo failed: rc = -114
[ 7475.319641] LustreError: Skipped 11 previous similar messages
[ 7478.850403] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7481.683167] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1356 to 0x2c0000400:1377
[ 7481.683818] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1612 to 0x280000400:1633
[ 7481.797870] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8385
[ 7481.802181] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8193
[ 7486.258506] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7487.557391] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7488.686627] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7495.192456] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 10:57:23 (1761317843)
[ 7502.385669] Lustre: lustre-MDT0001: Client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp) reconnecting
[ 7502.408727] Lustre: 221628:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@0000000033de1779 x1846867846658112/t34359738438(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:252/0 lens 560/448 e 0 to 0 dl 1761317857 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 7503.865252] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7505.195923] Lustre: Failing over lustre-MDT0000
[ 7505.197786] Lustre: Skipped 11 previous similar messages
[ 7505.324823] Lustre: server umount lustre-MDT0000 complete
[ 7505.327134] Lustre: Skipped 11 previous similar messages
[ 7506.406817] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 7506.428696] Lustre: Skipped 44 previous similar messages
[ 7513.568318] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317855/real 1761317855]  req@0000000082ef5b78 x1846867846807040/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761317862 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 7513.589108] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 24 previous similar messages
[ 7513.596596] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 7513.614294] LustreError: Skipped 6 previous similar messages
[ 7526.131850] LDISKFS-fs (dm-0): recovery complete
[ 7526.135383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7534.895822] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7536.734406] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8225
[ 7536.737759] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8417
[ 7542.244876] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7543.474625] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7549.896977] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 10:58:17 (1761317897)
[ 7550.628230] LustreError: 222172:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000c8836323 x1846867846675456/t34359738508(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:300/0 lens 560/448 e 0 to 0 dl 1761317905 ref 1 fl Interpret:/0/0 rc 0/0 job:'lfs.0'
[ 7550.645143] LustreError: 222172:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 7555.498734] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7567.328842] LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 7567.335287] LustreError: Skipped 330 previous similar messages
[ 7575.351119] LDISKFS-fs (dm-1): recovery complete
[ 7575.353457] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7575.517782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180
[ 7575.523349] Lustre: Skipped 12 previous similar messages
[ 7575.538518] Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect
[ 7575.541767] Lustre: Skipped 12 previous similar messages
[ 7577.501154] Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 7577.508437] Lustre: Skipped 12 previous similar messages
[ 7578.225390] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7580.650506] Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to  (at 0@lo)
[ 7580.653573] Lustre: Skipped 54 previous similar messages
[ 7580.696807] Lustre: 221627:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000c01ff73f x1846867846675456/t34359738508(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:330/0 lens 560/448 e 0 to 0 dl 1761317935 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 7580.700504] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1654 to 0x280000400:1697
[ 7580.700559] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1398 to 0x2c0000400:1441
[ 7584.521105] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7585.734281] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7592.429594] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 10:59:00 (1761317940)
[ 7599.621855] Lustre: lustre-MDT0001: Client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp) reconnecting
[ 7601.305435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7606.247473] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7627.516489] LDISKFS-fs (dm-0): recovery complete
[ 7627.519602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7634.919702] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283b075916 to 0x27c83d283b0770e0
[ 7634.935425] Lustre: Skipped 7 previous similar messages
[ 7638.477268] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7640.638130] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted.
[ 7640.648875] Lustre: Skipped 13 previous similar messages
[ 7640.682156] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8449
[ 7640.683084] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8257
[ 7645.725613] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7646.864452] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7670.645065] LDISKFS-fs (dm-1): recovery complete
[ 7670.646761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7674.601197] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7675.942589] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1452 to 0x2c0000400:1473
[ 7675.947169] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1708 to 0x280000400:1729
[ 7682.195193] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7683.089203] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7690.267508] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 11:00:38 (1761318038)
[ 7690.935932] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 7690.940870] Lustre: Skipped 3 previous similar messages
[ 7697.408067] Lustre: lustre-MDT0001: Client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp) reconnecting
[ 7697.423103] Lustre: 221628:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000008dc84cfe x1846867846712448/t42949673030(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:447/0 lens 560/448 e 0 to 0 dl 1761318052 ref 1 fl Interpret:/2/0 rc 0/0 job:'lfs.0'
[ 7697.437838] Lustre: 221628:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 7699.115058] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7704.254367] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7708.225432] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318056 with bad export cookie 2866608405819650272
[ 7708.231627] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 4 previous similar messages
[ 7726.935235] LDISKFS-fs (dm-0): recovery complete
[ 7726.939760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7743.918987] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7763.099942] LDISKFS-fs (dm-1): recovery complete
[ 7763.101988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7766.979897] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7769.278241] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8481
[ 7769.284402] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8289
[ 7769.318260] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1484 to 0x2c0000400:1505
[ 7774.545407] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7775.782662] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7777.046645] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7785.083693] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 11:02:12 (1761318132)
[ 7792.666604] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7793.125932] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7814.356798] LDISKFS-fs (dm-1): recovery complete
[ 7814.363171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7817.131538] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7819.797644] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1569
[ 7819.798674] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1761
[ 7823.320712] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7824.309163] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7829.316478] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 11:02:57 (1761318177)
[ 7830.058187] LustreError: 188912:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000ed00e82b x1846867846917824/t386547056673(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:579/0 lens 1488/4320 e 0 to 0 dl 1761318184 ref 1 fl Interpret:/0/0 rc 0/0 job:'osp_up0-1.0'
[ 7830.084954] LustreError: 188912:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 3 previous similar messages
[ 7835.338530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7856.457342] LDISKFS-fs (dm-0): recovery complete
[ 7856.459906] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7864.376690] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7866.412934] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8321
[ 7871.608297] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7872.783729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7879.599447] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 11:03:47 (1761318227)
[ 7885.217125] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7887.842232] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7889.389766] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7911.361195] LDISKFS-fs (dm-0): recovery complete
[ 7911.368852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7921.009104] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7922.817043] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8353
[ 7922.827317] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8513
[ 7929.322667] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 7931.025638] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7954.467786] LDISKFS-fs (dm-1): recovery complete
[ 7954.470091] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 7957.858798] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 7960.109568] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1793
[ 7960.109918] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1601
[ 7965.795179] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 7967.133198] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 7973.727190] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 11:05:21 (1761318321)
[ 7980.240943] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 7982.051392] Lustre: lustre-MDT0000: Received new MDS connection from 0@lo, keep former export from same NID
[ 7985.532488] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 7991.583470] LustreError: 7232:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318340 with bad export cookie 2866608405819660373
[ 7992.292869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping)
[ 7992.295438] Lustre: Skipped 5 previous similar messages
[ 8017.989774] LDISKFS-fs (dm-0): recovery complete
[ 8017.992888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8029.616335] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8050.417607] LDISKFS-fs (dm-1): recovery complete
[ 8050.420976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8054.353280] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8055.953859] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1825
[ 8055.960464] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1633
[ 8056.057688] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8545
[ 8062.074915] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8063.330319] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8064.562345] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8071.115743] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 11:06:58 (1761318418)
[ 8078.333133] Lustre: lustre-MDT0001: Client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp) reconnecting
[ 8078.344142] Lustre: 245238:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000004ffbe31c x1846867846765504/t60129542146(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:72/0 lens 496/2888 e 0 to 0 dl 1761318432 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0'
[ 8078.838132] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8102.306929] LDISKFS-fs (dm-0): recovery complete
[ 8102.309866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8109.630618] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8111.658167] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8385
[ 8111.661914] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8577
[ 8117.728838] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8119.128972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8126.612897] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 11:07:54 (1761318474)
[ 8133.948414] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8135.781748] Lustre: Failing over lustre-MDT0001
[ 8135.785734] Lustre: Skipped 12 previous similar messages
[ 8135.949139] Lustre: server umount lustre-MDT0001 complete
[ 8135.959534] Lustre: Skipped 12 previous similar messages
[ 8137.184646] LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107
[ 8137.186650] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 8137.193255] LustreError: Skipped 8 previous similar messages
[ 8137.196383] Lustre: Skipped 44 previous similar messages
[ 8158.549894] LDISKFS-fs (dm-1): recovery complete
[ 8158.552818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8162.355504] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8164.356855] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1665
[ 8164.359913] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1857
[ 8169.721785] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8171.079817] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8178.250203] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 11:08:45 (1761318525)
[ 8186.094097] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8187.398330] Lustre: lustre-MDT0001: Client a0f9a46b-d44e-44c6-8720-3bfb9d392d66 (at 192.168.203.35@tcp) reconnecting
[ 8187.401362] Lustre: Skipped 1 previous similar message
[ 8193.340533] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8197.646255] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.35@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 8197.658921] LustreError: Skipped 274 previous similar messages
[ 8207.328292] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761318548/real 1761318548]  req@000000000d55cfd0 x1846867847028032/t0(0) o400->MGC192.168.203.135@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761318555 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 8207.352367] Lustre: 3337:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 16 previous similar messages
[ 8207.366494] LustreError: 166-1: MGC192.168.203.135@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 8207.380216] LustreError: Skipped 6 previous similar messages
[ 8216.423986] LDISKFS-fs (dm-0): recovery complete
[ 8216.426944] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8224.740682] Lustre: MGC192.168.203.135@tcp: Connection restored to  (at 0@lo)
[ 8224.754160] Lustre: Skipped 51 previous similar messages
[ 8224.906691] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 8224.911888] Lustre: Skipped 12 previous similar messages
[ 8224.936123] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 8224.944505] Lustre: Skipped 12 previous similar messages
[ 8226.401790] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 8226.409103] Lustre: Skipped 12 previous similar messages
[ 8228.647860] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8230.488631] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8417
[ 8230.491847] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8609
[ 8237.501428] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8238.990316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8263.913314] LDISKFS-fs (dm-1): recovery complete
[ 8263.915775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8268.582280] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8269.292038] Lustre: lustre-MDT0001: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted.
[ 8269.297592] Lustre: Skipped 12 previous similar messages
[ 8269.315409] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1889
[ 8269.317945] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1697
[ 8276.452201] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8277.519425] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8284.489239] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 11:10:32 (1761318632)
[ 8285.470742] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 8285.488303] Lustre: Skipped 7 previous similar messages
[ 8290.975296] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8293.902379] Lustre: 246824:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000d5949318 x1846867846792256/t68719476738(0) o36->a0f9a46b-d44e-44c6-8720-3bfb9d392d66@192.168.203.35@tcp:288/0 lens 496/2888 e 0 to 0 dl 1761318648 ref 1 fl Interpret:/2/0 rc 0/0 job:'rmdir.0'
[ 8293.913678] Lustre: 246824:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 2 previous similar messages
[ 8297.058719] Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001
[ 8301.710972] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318650 with bad export cookie 2866608405819667653
[ 8301.720942] LustreError: 6221:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 3 previous similar messages
[ 8321.940270] LDISKFS-fs (dm-0): recovery complete
[ 8321.941984] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8334.757729] Lustre: Evicted from MGS (at 192.168.203.135@tcp) after server handle changed from 0x27c83d283b07b4c5 to 0x27c83d283b07be3b
[ 8334.779028] Lustre: Skipped 6 previous similar messages
[ 8338.987105] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8359.880410] LDISKFS-fs (dm-1): recovery complete
[ 8359.887702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8363.852720] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8366.092159] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8291 to 0x0:8641
[ 8366.093095] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8035 to 0x0:8449
[ 8366.122585] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1921
[ 8366.128383] Lustre: lustre-OST0001: deleting orphan objects from 0x2c0000400:1526 to 0x2c0000400:1729
[ 8372.323969] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid
[ 8373.676151] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8374.878546] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8381.637562] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 11:12:09 (1761318729)
[ 8383.055865] Lustre: 259113:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting a0f9a46b-d44e-44c6-8720-3bfb9d392d66 at adminstrative request
[ 8390.877456] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 11:12:18 (1761318738)
[ 8414.354706] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8417.576623] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8420.458116] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8500 to 0x0:8545
[ 8420.461689] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8693 to 0x0:8737
[ 8424.761196] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 8425.941141] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 8431.861289] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 11:12:59 (1761318779)
[ 8442.378207] Lustre: lustre-OST0000: Not available for connect from 192.168.203.35@tcp (stopping)
[ 8442.382814] Lustre: Skipped 7 previous similar messages
[ 8461.760585] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8463.257360] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8838 to 0x0:8865
[ 8463.260094] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1953
[ 8465.014799] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8473.303665] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 8474.847072] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 8482.192286] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 11:13:49 (1761318829)
[ 8490.740975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8494.033567] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8495.567405] Lustre: lustre-MDT0000: Denying connection for new client 34182a19-5c34-4700-b762-d24335668d22 (at 192.168.203.35@tcp), waiting for 1 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59
[ 8495.583418] Lustre: Skipped 1 previous similar message
[ 8496.138576] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8500 to 0x0:8577
[ 8496.139529] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8838 to 0x0:8897
[ 8504.503601] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 11:14:12 (1761318852)
[ 8509.855551] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 8530.629101] LDISKFS-fs (dm-2): recovery complete
[ 8530.631996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8531.996161] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8899 to 0x0:8929
[ 8532.000901] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:1985
[ 8533.491187] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8539.986708] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 8541.102766] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 8547.445250] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 11:14:55 (1761318895)
[ 8553.585333] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 8574.084026] LDISKFS-fs (dm-2): recovery complete
[ 8574.086471] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8575.438073] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.35@tcp inode [0x2000301a1:0x5:0x0] object 0x0:8930 extent [0-4194303]: client csum 4b91088c, server csum b27d04f6
[ 8575.516816] Lustre: lustre-OST0000: deleting orphan objects from 0x0:8931 to 0x0:8961
[ 8575.520576] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:2017
[ 8576.720727] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8582.812665] Lustre: DEBUG MARKER: oleg335-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 8584.048792] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 8589.875042] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 11:15:37 (1761318937)
[ 8594.771734] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 8599.456911] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 8606.178411] LustreError: 8224:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761318954 with bad export cookie 2866608405819694330
[ 8606.189424] LustreError: 8224:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) Skipped 2 previous similar messages
[ 8625.694065] LDISKFS-fs (dm-0): recovery complete
[ 8625.696572] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8641.218959] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8644.427553] Lustre: lustre-OST0001: deleting orphan objects from 0x0:8500 to 0x0:8609
[ 8659.817603] LDISKFS-fs (dm-2): recovery complete
[ 8659.827215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8661.600967] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:2049
[ 8663.085414] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8674.023053] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 11:17:01 (1761319021)
[ 8700.689467] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
[ 8712.914709] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8719.015227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
[ 8721.481801] Lustre: DEBUG MARKER: oleg335-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 8722.758107] Lustre: lustre-OST0000: Denying connection for new client f63854db-42df-4dc1-bdc5-c6434ef35146 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:07
[ 8728.063248] Lustre: lustre-OST0000: Denying connection for new client f63854db-42df-4dc1-bdc5-c6434ef35146 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:01
[ 8738.304633] Lustre: lustre-OST0000: Denying connection for new client f63854db-42df-4dc1-bdc5-c6434ef35146 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:51
[ 8738.311543] Lustre: Skipped 1 previous similar message
[ 8758.787679] Lustre: lustre-OST0000: Denying connection for new client f63854db-42df-4dc1-bdc5-c6434ef35146 (at 192.168.203.35@tcp), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 0:31
[ 8758.799757] Lustre: Skipped 3 previous similar messages
[ 8790.000977] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 8790.009337] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 8790.025977] Lustre: lustre-OST0000: deleting orphan objects from 0x280000400:1740 to 0x280000400:2081
[ 8790.027085] Lustre: lustre-OST0000: deleting orphan objects from 0x0:9012 to 0x0:9033
[ 8796.283914] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 68 sec
[ 8802.668586] Lustre: DEBUG MARKER: free_before: 7646580 free_after: 7646580
[ 8806.057223] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 11:19:14 (1761319154)
[ 8898.018208] Lustre: DEBUG MARKER: replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed