[ 2617.851199] LustreError: 90636:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 2617.855165] LustreError: 90636:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 2617.860656] Lustre: 90681:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 2617.864276] Lustre: 90681:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 2617.867777] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 2617.940525] Lustre: lustre-OST0000: deleting orphan objects from 0x0:1635 to 0x0:1697
[ 2617.944849] Lustre: lustre-OST0001: deleting orphan objects from 0x0:1539 to 0x0:1633
[ 2620.552110] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2630.794388] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:54:56 (1761314096)
[ 2651.192965] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2670.649142] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 2670.655908] Lustre: Skipped 25 previous similar messages
[ 2673.329753] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2678.890081] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2034 to 0x0:2049
[ 2678.890305] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2098 to 0x0:2113
[ 2682.423686] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2683.311548] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2698.888924] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 09:56:04 (1761314164)
[ 2715.843770] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2754.811395] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2757.182159] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2514 to 0x0:2529
[ 2757.182554] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2465
[ 2763.870547] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2765.839647] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2783.183553] Lustre: DEBUG MARKER: == replay-single test 40: cause recovery in ptlrpc, ensure IO continues ========================================================== 09:57:28 (1761314248)
[ 2784.638215] Lustre: DEBUG MARKER: SKIP: replay-single test_40 layout_lock needs MDS connection for IO
[ 2786.481644] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 09:57:32 (1761314252)
[ 2788.537351] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request
[ 2789.407608] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2789.411034] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 2789.422901] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2450 to 0x0:2497
[ 2794.844585] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 09:57:40 (1761314260)
[ 2811.915495] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 2820.140824] Lustre: lustre-OST0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 2823.649325] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 2823.667811] LustreError: Skipped 1 previous similar message
[ 2840.320723] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:2977
[ 2842.039803] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2895.373216] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 09:59:21 (1761314361)
[ 2898.385264] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 2912.096447] Lustre: 2994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761314372/real 1761314372]  req@00000000c0343b46 x1846868988010496/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761314379 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 2912.108693] Lustre: 2994:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 121 previous similar messages
[ 2929.640634] Lustre: MGC192.168.203.141@tcp: Connection restored to 192.168.203.141@tcp (at 0@lo)
[ 2929.652936] Lustre: Skipped 42 previous similar messages
[ 2930.118553] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 2930.122138] Lustre: Skipped 12 previous similar messages
[ 2931.939924] Lustre: *** cfs_fail_loc=204, val=2147483648***
[ 2931.939954] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2913
[ 2933.624927] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 2939.363306] LustreError: 98776:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -11
[ 2939.364692] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 2940.385358] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3009
[ 2941.232985] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 2942.428555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 2959.221383] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 10:00:25 (1761314425)
[ 2962.648907] LustreError: 98756:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 2968.032092] LustreError: 98756:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 2968.038655] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 2968.084733] LustreError: 6315:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 2969.583646] LustreError: 98755:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 2974.689082] LustreError: 98755:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 2974.696466] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 2976.085066] LustreError: 98756:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 2981.344162] LustreError: 98756:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 2981.346994] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 2982.601985] LustreError: 98757:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 2988.000112] LustreError: 98757:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 2988.026415] LustreError: 98757:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 2988.032908] LustreError: 98757:0:(libcfs_fail.h:180:cfs_race()) Skipped 1 previous similar message
[ 2989.522422] LustreError: 100119:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 2994.656328] LustreError: 100119:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 2994.671139] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 2994.686130] Lustre: Skipped 2 previous similar messages
[ 3002.808048] LustreError: 100119:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3002.813402] LustreError: 100119:0:(libcfs_fail.h:169:cfs_race()) Skipped 1 previous similar message
[ 3007.968197] LustreError: 100119:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3007.976290] LustreError: 100119:0:(libcfs_fail.h:178:cfs_race()) Skipped 1 previous similar message
[ 3007.997293] LustreError: 100119:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3015.136161] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 3015.147622] Lustre: Skipped 3 previous similar messages
[ 3015.188330] LustreError: 98757:0:(libcfs_fail.h:180:cfs_race()) cfs_fail_race id 701 waking
[ 3023.151266] LustreError: 98756:0:(libcfs_fail.h:169:cfs_race()) cfs_race id 701 sleeping
[ 3023.155174] LustreError: 98756:0:(libcfs_fail.h:169:cfs_race()) Skipped 2 previous similar messages
[ 3028.448145] LustreError: 98756:0:(libcfs_fail.h:178:cfs_race()) cfs_fail_race id 701 awake: rc=0
[ 3028.452710] LustreError: 98756:0:(libcfs_fail.h:178:cfs_race()) Skipped 2 previous similar messages
[ 3034.146281] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 10:01:40 (1761314500)
[ 3035.411464] LustreError: 99456:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms
[ 3041.828048] Lustre: lustre-MDT0000: Export 000000002937d3ee already connecting from 192.168.203.41@tcp
[ 3043.306420] Lustre: lustre-MDT0000: Export 000000002937d3ee already connecting from 192.168.203.41@tcp
[ 3044.817087] Lustre: lustre-MDT0000: Export 000000002937d3ee already connecting from 192.168.203.41@tcp
[ 3048.101208] Lustre: lustre-MDT0000: Export 000000002937d3ee already connecting from 192.168.203.41@tcp
[ 3048.110033] Lustre: Skipped 1 previous similar message
[ 3052.948134] Lustre: lustre-MDT0000: Export 000000002937d3ee already connecting from 192.168.203.41@tcp
[ 3052.952610] Lustre: Skipped 2 previous similar messages
[ 3057.353244] LustreError: 99456:0:(fail.c:144:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted
[ 3057.366337] Lustre: 99456:0:(service.c:2348:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/2s); client may timeout  req@000000003f509715 x1846868978407808/t0(0) o38->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:0/0 lens 520/416 e 0 to 0 dl 1761314522 ref 1 fl Complete:H/0/0 rc 0/0 job:'lctl.0'
[ 3061.281886] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 3061.294076] Lustre: Skipped 4 previous similar messages
[ 3061.588804] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 10:02:07 (1761314527)
[ 3064.699037] LustreError: 102430:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3064.703317] LustreError: 102430:0:(osd_handler.c:698:osd_ro()) Skipped 9 previous similar messages
[ 3065.453911] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3068.639773] Lustre: Failing over lustre-MDT0000
[ 3068.643957] Lustre: Skipped 11 previous similar messages
[ 3068.959857] Lustre: server umount lustre-MDT0000 complete
[ 3068.965744] Lustre: Skipped 11 previous similar messages
[ 3074.907552] LustreError: 166-1: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3074.916726] LustreError: Skipped 9 previous similar messages
[ 3074.921054] Lustre: Evicted from MGS (at 192.168.203.141@tcp) after server handle changed from 0x450ee28f1ddbd414 to 0x450ee28f1ddbdd13
[ 3074.928476] Lustre: Skipped 9 previous similar messages
[ 3075.108356] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3075.119859] Lustre: *** cfs_fail_loc=712, val=0***
[ 3075.122734] Lustre: Skipped 23 previous similar messages
[ 3075.141290] LustreError: 36182:0:(service.c:1226:ptlrpc_check_req()) @@@ Invalid replay without recovery  req@00000000f8a8ecb4 x1846868988039552/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0'
[ 3075.158032] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
[ 3075.333646] LustreError: 103052:0:(mdt_handler.c:7428:mdt_iocontrol()) lustre-MDT0000: Aborting recovery for device
[ 3075.339353] LustreError: 103052:0:(ldlm_lib.c:2882:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
[ 3075.344986] Lustre: 103100:0:(ldlm_lib.c:2288:target_recovery_overseer()) recovery is aborted, evict exports in recovery
[ 3075.359915] Lustre: 103100:0:(ldlm_lib.c:2288:target_recovery_overseer()) Skipped 2 previous similar messages
[ 3075.369252] Lustre: lustre-MDT0000: disconnecting 1 stale clients
[ 3075.443276] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2945
[ 3075.447346] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3041
[ 3079.035578] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3085.794184] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping)
[ 3108.469695] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3110.435839] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3110.446247] Lustre: Skipped 6 previous similar messages
[ 3110.519874] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3110.524964] Lustre: Skipped 6 previous similar messages
[ 3110.579281] Lustre: lustre-OST0000: deleting orphan objects from 0x0:2931 to 0x0:3073
[ 3110.579822] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2898 to 0x0:2977
[ 3116.818420] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3118.267018] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3125.313445] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 10:03:11 (1761314591)
[ 3125.407970] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 3131.216403] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 10:03:17 (1761314597)
[ 3131.806982] Lustre: *** cfs_fail_loc=122, val=2147483648***
[ 3131.812416] LustreError: 104074:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000c315fca7 x1846868978430208/t0(0) o700->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:20/0 lens 264/248 e 0 to 0 dl 1761314605 ref 1 fl Interpret:/0/0 rc 0/0 job:'touch.0'
[ 3160.420963] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3167.863925] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3075 to 0x0:3105
[ 3167.867571] Lustre: lustre-OST0001: deleting orphan objects from 0x0:2979 to 0x0:3009
[ 3172.654156] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3173.997628] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3181.192400] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:04:07 (1761314647)
[ 3183.584512] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 3200.607197] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3116 to 0x0:3137
[ 3201.575740] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3207.686929] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 3208.792531] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 3278.353122] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 10:05:44 (1761314744)
[ 3281.208945] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3299.455277] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3299.457637] Lustre: Skipped 9 previous similar messages
[ 3301.872537] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3309.539707] Lustre: *** cfs_fail_loc=216, val=0***
[ 3309.541771] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3030 to 0x0:3073
[ 3309.544348] LustreError: 108938:0:(osp_precreate.c:967:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -30
[ 3310.560798] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3148 to 0x0:3169
[ 3376.670472] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 10:07:22 (1761314842)
[ 3378.331359] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting
[ 3378.334516] Lustre: Skipped 2 previous similar messages
[ 3378.343516] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3201
[ 3379.089994] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3233
[ 3388.955990] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 10:07:34 (1761314854)
[ 3419.744425] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3420.709631] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 3420.713330] LustreError: 110625:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@0000000046efaefb x1846868978479232/t0(0) o101->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:309/0 lens 328/344 e 0 to 0 dl 1761314894 ref 1 fl Complete:/40/0 rc 0/0 job:'ldlm_lock_repla.0'
[ 3427.870073] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 3427.965540] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3085 to 0x0:3105
[ 3427.965544] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3180 to 0x0:3265
[ 3432.207869] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3433.283880] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3440.963543] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 10:08:26 (1761314906)
[ 3442.838048] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3445.978964] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3477.170171] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3482.660900] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3267 to 0x0:3297
[ 3486.071274] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3487.132055] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3492.384144] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 10:09:18 (1761314958)
[ 3493.034803] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3496.887151] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3517.457811] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 3517.462322] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3085 to 0x0:3137
[ 3518.203155] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3523.704343] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3524.582739] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3530.133076] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 10:09:56 (1761314996)
[ 3530.894504] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3535.038932] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3546.080334] Lustre: 2997:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315006/real 1761315006]  req@000000001b7ec0da x1846868988110464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761315013 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:0.0'
[ 3546.099873] Lustre: 2997:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 53 previous similar messages
[ 3563.499235] Lustre: MGC192.168.203.141@tcp: Connection restored to  (at 0@lo)
[ 3563.501718] Lustre: Skipped 28 previous similar messages
[ 3563.944637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 3563.948620] Lustre: Skipped 8 previous similar messages
[ 3567.379192] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3570.762265] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 3570.764299] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3085 to 0x0:3169
[ 3578.915369] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 10:10:44 (1761315044)
[ 3580.932426] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3580.934258] Lustre: *** cfs_fail_loc=13b, val=2147483648***
[ 3580.936157] LustreError: 115897:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@000000002acb1f87 x1846868978502848/t261993005073(0) o35->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:469/0 lens 392/456 e 0 to 0 dl 1761315054 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 3612.455652] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3621.949867] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3171 to 0x0:3201
[ 3621.951134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3393
[ 3621.975446] Lustre: 117245:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000005e196b83 x1846868978502848/t261993005073(0) o35->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:510/0 lens 392/456 e 0 to 0 dl 1761315095 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 3626.390786] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3627.798862] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3634.904546] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 10:11:40 (1761315100)
[ 3635.744297] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3635.746224] LustreError: 117241:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000b0d71bf1 x1846868978509760/t266287972368(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:524/0 lens 504/448 e 0 to 0 dl 1761315109 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 3640.276456] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3660.321428] Lustre: 118987:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@000000001f7337ef x1846868978509760/t266287972368(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:548/0 lens 504/448 e 0 to 0 dl 1761315133 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 3660.331338] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3233
[ 3660.331977] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3331 to 0x0:3425
[ 3660.778934] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3667.626228] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3668.954789] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3675.399631] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 10:12:21 (1761315141)
[ 3676.157281] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3676.159808] LustreError: 118985:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000971b68d6 x1846868978516736/t270582939664(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:564/0 lens 504/448 e 0 to 0 dl 1761315149 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 3677.772490] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3679.460656] LustreError: 120120:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 3679.466043] LustreError: 120120:0:(osd_handler.c:698:osd_ro()) Skipped 5 previous similar messages
[ 3680.034681] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3681.158150] Lustre: Failing over lustre-MDT0000
[ 3681.160067] Lustre: Skipped 10 previous similar messages
[ 3681.365441] Lustre: server umount lustre-MDT0000 complete
[ 3681.371676] Lustre: Skipped 10 previous similar messages
[ 3692.001299] LustreError: 166-1: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 3692.002596] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 3692.010101] LustreError: Skipped 9 previous similar messages
[ 3692.024886] Lustre: Skipped 22 previous similar messages
[ 3698.148098] Lustre: Evicted from MGS (at 192.168.203.141@tcp) after server handle changed from 0x450ee28f1ddc21e6 to 0x450ee28f1ddc27ea
[ 3698.157672] Lustre: Skipped 9 previous similar messages
[ 3700.855920] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3265
[ 3700.858983] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3427 to 0x0:3457
[ 3700.870056] Lustre: 120719:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000971b68d6 x1846868978516736/t270582939664(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:589/0 lens 504/448 e 0 to 0 dl 1761315174 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 3701.002608] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3708.982985] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 10:12:55 (1761315175)
[ 3709.634831] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 3709.642779] Lustre: Skipped 1 previous similar message
[ 3709.644787] LustreError: 120719:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000d85f9795 x1846868978523136/t274877906960(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:598/0 lens 504/448 e 0 to 0 dl 1761315183 ref 1 fl Interpret:/0/0 rc 0/0 job:'mcreate.0'
[ 3709.654666] LustreError: 120719:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 3711.177434] Lustre: *** cfs_fail_loc=115, val=2147483648***
[ 3711.179119] Lustre: Skipped 1 previous similar message
[ 3714.293513] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3733.470884] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 3733.475200] Lustre: Skipped 10 previous similar messages
[ 3733.584476] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 3733.590069] Lustre: Skipped 10 previous similar messages
[ 3733.641330] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3489
[ 3733.646144] Lustre: 122304:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000848eb5ab x1846868978523136/t274877906960(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:621/0 lens 504/448 e 0 to 0 dl 1761315206 ref 1 fl Interpret:/2/0 rc 0/0 job:'mcreate.0'
[ 3733.652568] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3203 to 0x0:3297
[ 3733.689360] Lustre: 122304:0:(mdt_recovery.c:200:mdt_req_from_lrd()) Skipped 1 previous similar message
[ 3736.078119] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3745.635218] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 10:13:31 (1761315211)
[ 3746.546768] Lustre: *** cfs_fail_loc=107, val=2147483648***
[ 3748.399762] Lustre: *** cfs_fail_loc=13b, val=315***
[ 3748.402472] LustreError: 122307:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000fc0a9fc7 x1846868978529408/t279172874256(0) o35->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:636/0 lens 392/456 e 0 to 0 dl 1761315221 ref 1 fl Interpret:/0/0 rc 0/0 job:'multiop.0'
[ 3752.522315] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3752.990075] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnecting
[ 3752.995619] Lustre: Skipped 1 previous similar message
[ 3753.011141] Lustre: 122307:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000694d9d67 x1846868978529408/t279172874256(0) o35->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:641/0 lens 392/456 e 0 to 0 dl 1761315226 ref 1 fl Interpret:/2/0 rc 0/0 job:'multiop.0'
[ 3771.591352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3521
[ 3771.594126] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3299 to 0x0:3329
[ 3774.015094] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3781.799426] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 10:14:07 (1761315247)
[ 3782.443462] Lustre: *** cfs_fail_loc=12b, val=2147483991***
[ 3782.447248] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 3782.449864] Lustre: Skipped 1 previous similar message
[ 3789.861724] Lustre: 123804:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000d6cb1229 x1846868978534272/t283467841549(0) o101->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:678/0 lens 664/3424 e 0 to 0 dl 1761315263 ref 1 fl Interpret:/2/0 rc 0/0 job:'touch.0'
[ 3794.209911] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 10:14:20 (1761315260)
[ 3796.754375] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3820.837864] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3824.764938] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3553
[ 3824.768335] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3331 to 0x0:3361
[ 3829.094934] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3830.456827] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3847.897147] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 10:15:13 (1761315313)
[ 3851.853418] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3884.822869] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3895.412403] Lustre: lustre-OST0000: deleting orphan objects from 0x0:3459 to 0x0:3585
[ 3895.414881] Lustre: lustre-OST0001: deleting orphan objects from 0x0:3363 to 0x0:3393
[ 3899.276095] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3900.485215] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 3904.222196] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475
[ 3912.935496] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 10:16:19 (1761315379)
[ 3952.998660] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 3956.777026] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 3956.780677] Lustre: Skipped 1 previous similar message
[ 3983.238680] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 3983.241391] Lustre: Skipped 11 previous similar messages
[ 3986.153254] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 3988.543927] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4865
[ 3988.546644] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4644 to 0x0:4673
[ 3992.730258] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 3993.814769] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4054.274376] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 10:18:40 (1761315520)
[ 4058.181767] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4059.692392] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 4087.076058] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4091.515238] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4836 to 0x0:4897
[ 4091.516645] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4675 to 0x0:4705
[ 4096.598699] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4097.916895] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4106.103224] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid
[ 4107.590494] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec
[ 4112.909968] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 10:19:38 (1761315578)
[ 4119.065087] Lustre: *** cfs_fail_loc=123, val=2147483648***
[ 4163.594603] Lustre: *** cfs_fail_loc=119, val=2147483648***
[ 4163.603195] LustreError: 131446:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000d8dd7ce8 x1846868980010048/t300647710728(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:331/0 lens 66040/440 e 0 to 0 dl 1761315671 ref 1 fl Interpret:/0/0 rc 0/0 job:'setfattr.0'
[ 4163.631969] LustreError: 131446:0:(ldlm_lib.c:3224:target_send_reply_msg()) Skipped 1 previous similar message
[ 4206.660604] Lustre: 131999:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000706f43e4 x1846868980010048/t300647710728(0) o36->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:375/0 lens 66040/440 e 0 to 0 dl 1761315715 ref 1 fl Interpret:/2/0 rc 0/0 job:'setfattr.0'
[ 4214.884351] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59
[ 4216.327317] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 10:21:22 (1761315682)
[ 4225.550238] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4239.845388] Lustre: 2997:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761315700/real 1761315700]  req@00000000acc1c400 x1846868988546624/t0(0) o400->MGC192.168.203.141@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1761315707 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:2.0'
[ 4239.863530] Lustre: 2997:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 75 previous similar messages
[ 4246.000593] Lustre: MGC192.168.203.141@tcp: Connection restored to  (at 0@lo)
[ 4246.007321] Lustre: Skipped 29 previous similar messages
[ 4246.339825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 4246.343613] Lustre: Skipped 9 previous similar messages
[ 4248.505286] Lustre: lustre-OST0001: deleting orphan objects from 0x0:4807 to 0x0:4833
[ 4248.507139] Lustre: lustre-OST0000: deleting orphan objects from 0x0:4998 to 0x0:5025
[ 4249.754500] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4257.506597] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4258.742451] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4267.620905] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 10:22:13 (1761315733)
[ 4283.638663] LustreError: 135220:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 4283.643969] LustreError: 135220:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages
[ 4284.282179] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 4294.380757] Lustre: Failing over lustre-OST0000
[ 4294.383171] Lustre: Skipped 7 previous similar messages
[ 4294.415031] Lustre: server umount lustre-OST0000 complete
[ 4294.419076] Lustre: Skipped 7 previous similar messages
[ 4294.741753] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107
[ 4294.745087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 4294.749816] Lustre: Skipped 15 previous similar messages
[ 4294.752944] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4294.760575] LustreError: Skipped 13 previous similar messages
[ 4315.095854] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5441
[ 4316.073829] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4330.466408] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 4330.469710] LustreError: Skipped 7 previous similar messages
[ 4348.838698] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect
[ 4348.842115] Lustre: Skipped 7 previous similar messages
[ 4349.428022] Lustre: lustre-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted.
[ 4349.437052] Lustre: Skipped 7 previous similar messages
[ 4349.450106] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5473
[ 4350.872761] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4359.781940] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4361.181973] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4399.459299] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 10:24:25 (1761315865)
[ 4410.851576] LustreError: 166-1: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 4410.865335] LustreError: Skipped 7 previous similar messages
[ 4428.257597] Lustre: Evicted from MGS (at 192.168.203.141@tcp) after server handle changed from 0x450ee28f1de07719 to 0x450ee28f1de182d2
[ 4428.267909] Lustre: Skipped 7 previous similar messages
[ 4429.507354] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5249
[ 4429.510907] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5505
[ 4432.665123] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4472.872155] LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 4472.879767] LustreError: Skipped 15 previous similar messages
[ 4474.383641] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5234 to 0x0:5281
[ 4474.390758] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5426 to 0x0:5537
[ 4476.574935] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4484.534828] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4485.855807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4493.378576] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 10:25:59 (1761315959)
[ 4525.742394] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5569
[ 4527.130148] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4535.618405] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 4536.809752] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 4545.490514] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 10:26:51 (1761316011)
[ 4554.192235] Lustre: *** cfs_fail_loc=605, val=0***
[ 4554.194112] LustreError: 142508:0:(llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=0000000038811fb1 failed: rc = -95
[ 4554.199321] LustreError: 142508:0:(obd_config.c:774:class_setup()) setup MGS failed (-95)
[ 4554.202288] LustreError: 142508:0:(obd_mount.c:200:lustre_start_simple()) MGS setup error -95
[ 4554.205523] LustreError: 142508:0:(obd_mount_server.c:131:server_deregister_mount()) MGS not registered
[ 4554.208620] LustreError: 15e-a: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded?
[ 4554.212183] LustreError: 142508:0:(obd_mount_server.c:1644:server_put_super()) no obd lustre-MDT0000
[ 4554.299563] LustreError: 142508:0:(super25.c:183:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -95
[ 4563.369625] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5283 to 0x0:5313
[ 4563.374770] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5539 to 0x0:5601
[ 4566.707310] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4573.854022] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 10:27:19 (1761316039)
[ 4577.009275] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 4607.651500] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 4607.654252] Lustre: Skipped 8 previous similar messages
[ 4611.128913] Lustre: *** cfs_fail_loc=707, val=0***
[ 4611.726241] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 4653.104559] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnected, waiting for 1 clients in recovery for 0:57
[ 4653.600827] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5614 to 0x0:5633
[ 4653.601268] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5327 to 0x0:5345
[ 4658.951457] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 4660.194767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 4668.038409] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 10:28:54 (1761316134)
[ 4695.062099] LustreError: 144211:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 6000ms
[ 4701.096169] LustreError: 144211:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4716.027176] Lustre: DEBUG MARKER: == replay-single test 65b: AT: verify early replies on packed reply / bulk ========================================================== 10:29:42 (1761316182)
[ 4743.489931] LustreError: 34933:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 sleeping for 6000ms
[ 4749.528122] LustreError: 34933:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 224 awake
[ 4756.962896] Lustre: DEBUG MARKER: == replay-single test 66a: AT: verify MDT service time adjusts with no early replies ========================================================== 10:30:22 (1761316222)
[ 4783.266219] LustreError: 144213:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 5000ms
[ 4788.304263] LustreError: 144213:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4790.273571] LustreError: 144667:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 10000ms
[ 4800.296124] LustreError: 144667:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4815.442728] Lustre: DEBUG MARKER: == replay-single test 66b: AT: verify net latency adjusts ========================================================== 10:31:21 (1761316281)
[ 4868.929281] Lustre: DEBUG MARKER: == replay-single test 67a: AT: verify slow request processing doesn't induce reconnects ========================================================== 10:32:14 (1761316334)
[ 4896.692615] LustreError: 147332:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 4897.112126] LustreError: 147332:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4904.833144] LustreError: 144681:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 4904.846150] LustreError: 144681:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 20 previous similar messages
[ 4905.288154] LustreError: 144681:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4905.295833] LustreError: 144681:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 20 previous similar messages
[ 4921.234571] LustreError: 144667:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 400ms
[ 4921.239207] LustreError: 144667:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 48 previous similar messages
[ 4921.665216] LustreError: 144667:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake
[ 4921.681993] LustreError: 144667:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 48 previous similar messages
[ 4928.546500] Lustre: DEBUG MARKER: == replay-single test 67b: AT: verify instant slowdown doesn't induce reconnects ========================================================== 10:33:14 (1761316394)
[ 4958.636633] Lustre: DEBUG MARKER: phase 2
[ 4966.251954] Lustre: DEBUG MARKER: == replay-single test 68: AT: verify slowing locks ======= 10:33:52 (1761316432)
[ 5045.658084] Lustre: DEBUG MARKER: == replay-single test 70a: check multi client t-f ======== 10:35:11 (1761316511)
[ 5046.928936] Lustre: DEBUG MARKER: SKIP: replay-single test_70a Need two or more clients, have 1
[ 5048.490230] Lustre: DEBUG MARKER: == replay-single test 70b: dbench 1mdts recovery; 1 clients ========================================================== 10:35:14 (1761316514)
[ 5051.392252] Lustre: DEBUG MARKER: Started rundbench load pid=131138 ...
[ 5055.023397] LustreError: 151018:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only ***
[ 5055.027034] LustreError: 151018:0:(osd_handler.c:698:osd_ro()) Skipped 1 previous similar message
[ 5055.987765] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5058.236222] Lustre: DEBUG MARKER: test_70b fail mds1 1 times
[ 5059.649712] Lustre: Failing over lustre-MDT0000
[ 5059.651524] Lustre: Skipped 6 previous similar messages
[ 5059.709687] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 5059.712697] Lustre: Skipped 1 previous similar message
[ 5060.037446] Lustre: server umount lustre-MDT0000 complete
[ 5060.042548] Lustre: Skipped 7 previous similar messages
[ 5070.242130] Lustre: 2994:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761316530/real 1761316530]  req@00000000c89cf660 x1846868988728896/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761316537 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:1.0'
[ 5070.255480] Lustre: 2994:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 66 previous similar messages
[ 5070.262626] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5070.270071] Lustre: Skipped 10 previous similar messages
[ 5070.278697] LustreError: 166-1: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5070.287600] LustreError: Skipped 3 previous similar messages
[ 5086.689555] Lustre: Evicted from MGS (at 192.168.203.141@tcp) after server handle changed from 0x450ee28f1de19341 to 0x450ee28f1de1e6a7
[ 5086.694214] Lustre: Skipped 3 previous similar messages
[ 5086.701820] Lustre: MGC192.168.203.141@tcp: Connection restored to  (at 0@lo)
[ 5086.706934] Lustre: Skipped 17 previous similar messages
[ 5087.152267] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5087.158085] Lustre: Skipped 7 previous similar messages
[ 5090.900982] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5098.653539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5098.675189] Lustre: Skipped 5 previous similar messages
[ 5099.256177] Lustre: lustre-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5099.259683] Lustre: Skipped 5 previous similar messages
[ 5099.298737] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5424 to 0x0:5441
[ 5099.299053] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5734 to 0x0:5761
[ 5103.776558] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5105.097787] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5110.937120] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5113.012844] Lustre: DEBUG MARKER: test_70b fail mds1 2 times
[ 5114.200627] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 5135.933280] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5146.427566] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5783 to 0x0:5825
[ 5146.427588] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5464 to 0x0:5505
[ 5150.156854] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5151.265761] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5156.545435] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5158.564036] Lustre: DEBUG MARKER: test_70b fail mds1 3 times
[ 5178.922468] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5188.741620] Lustre: lustre-OST0001: deleting orphan objects from 0x0:5548 to 0x0:5569
[ 5192.875548] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5194.015756] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5200.326104] Lustre: DEBUG MARKER: == replay-single test 70c: tar 1mdts recovery ============ 10:37:46 (1761316666)
[ 5325.602136] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5337.691810] Lustre: DEBUG MARKER: test_70c fail mds1 1 times
[ 5365.141570] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 5365.145452] Lustre: Skipped 3 previous similar messages
[ 5368.402031] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5374.732772] Lustre: lustre-OST0001: deleting orphan objects from 0x0:6958 to 0x0:6977
[ 5374.736134] Lustre: lustre-OST0000: deleting orphan objects from 0x0:5869 to 0x0:5889
[ 5379.986815] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5381.541807] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5458.646934] Lustre: DEBUG MARKER: == replay-single test 70d: mkdir/rmdir striped dir 1mdts recovery ========================================================== 10:42:04 (1761316924)
[ 5459.853280] Lustre: DEBUG MARKER: SKIP: replay-single test_70d needs >= 2 MDTs
[ 5461.277359] Lustre: DEBUG MARKER: == replay-single test 70e: rename cross-MDT with random fails ========================================================== 10:42:07 (1761316927)
[ 5462.618559] Lustre: DEBUG MARKER: SKIP: replay-single test_70e needs >= 2 MDTs
[ 5464.114983] Lustre: DEBUG MARKER: == replay-single test 70f: OSS O_DIRECT recovery with 1 clients ========================================================== 10:42:10 (1761316930)
[ 5470.930978] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5473.158481] Lustre: DEBUG MARKER: test_70f failing OST 1 times
[ 5475.884346] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5475.890910] LustreError: Skipped 6 previous similar messages
[ 5477.348290] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5491.656352] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6128 to 0x0:6145
[ 5492.944872] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5500.071801] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5501.376561] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5511.946917] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5514.335321] Lustre: DEBUG MARKER: test_70f failing OST 2 times
[ 5516.853726] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5516.878557] LustreError: Skipped 6 previous similar messages
[ 5517.280607] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5536.635336] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6128 to 0x0:6177
[ 5540.567522] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5551.807668] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5553.483289] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5563.361524] Lustre: DEBUG MARKER: == replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery ========================================================== 10:43:49 (1761317029)
[ 5564.641251] Lustre: DEBUG MARKER: SKIP: replay-single test_71a needs >= 2 MDTs
[ 5566.208862] Lustre: DEBUG MARKER: == replay-single test 73a: open(O_CREAT), unlink, replay, reconnect before open replay, close ========================================================== 10:43:51 (1761317031)
[ 5569.275866] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5594.431773] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5597.739701] Lustre: *** cfs_fail_loc=302, val=2147483648***
[ 5604.941198] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 5605.082362] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6179 to 0x0:6209
[ 5605.082511] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7217 to 0x0:7233
[ 5610.532455] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5611.882453] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5619.519973] Lustre: DEBUG MARKER: == replay-single test 73b: open(O_CREAT), unlink, replay, reconnect at open_replay reply, close ========================================================== 10:44:44 (1761317084)
[ 5623.089496] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5652.038664] Lustre: *** cfs_fail_loc=157, val=2147483648***
[ 5652.045877] LustreError: 163395:0:(ldlm_lib.c:3224:target_send_reply_msg()) @@@ dropping reply  req@00000000c2029d48 x1846868984678656/t343597383683(343597383683) o101->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:275/0 lens 592/600 e 0 to 0 dl 1761317125 ref 1 fl Interpret:/4/0 rc 301/0 job:'multiop.0'
[ 5653.895921] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5659.183287] Lustre: lustre-MDT0000: Client 6ffc8e48-4b56-455b-9c28-74b2e24b21e1 (at 192.168.203.41@tcp) reconnected, waiting for 1 clients in recovery for 0:58
[ 5659.216714] Lustre: 163394:0:(mdt_recovery.c:200:mdt_req_from_lrd()) @@@ restoring transno  req@00000000f0f76ce8 x1846868984678656/t343597383683(343597383683) o101->6ffc8e48-4b56-455b-9c28-74b2e24b21e1@192.168.203.41@tcp:282/0 lens 592/3424 e 0 to 0 dl 1761317132 ref 1 fl Interpret:/6/0 rc 0/0 job:'multiop.0'
[ 5659.345627] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7235 to 0x0:7265
[ 5659.345933] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6179 to 0x0:6241
[ 5663.617237] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5664.686985] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5672.210741] Lustre: DEBUG MARKER: == replay-single test 74: Ensure applications don't fail waiting for OST recovery ========================================================== 10:45:38 (1761317138)
[ 5674.712566] Lustre: Failing over lustre-OST0000
[ 5674.721616] Lustre: Skipped 7 previous similar messages
[ 5674.807415] Lustre: server umount lustre-OST0000 complete
[ 5674.809445] Lustre: Skipped 7 previous similar messages
[ 5674.977708] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5674.986784] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
[ 5674.998601] Lustre: Skipped 13 previous similar messages
[ 5675.002184] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5675.013595] LustreError: Skipped 7 previous similar messages
[ 5689.248545] Lustre: 2995:0:(client.c:2295:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1761317149/real 1761317149]  req@0000000066235d81 x1846868989246272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1761317156 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u8:3.0'
[ 5689.270731] Lustre: 2995:0:(client.c:2295:ptlrpc_expire_one_request()) Skipped 33 previous similar messages
[ 5690.343240] LustreError: 166-1: MGC192.168.203.141@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
[ 5690.346906] LustreError: Skipped 5 previous similar messages
[ 5696.424299] Lustre: Evicted from MGS (at 192.168.203.141@tcp) after server handle changed from 0x450ee28f1dea1e6d to 0x450ee28f1dea2256
[ 5696.441939] Lustre: Skipped 5 previous similar messages
[ 5696.443951] Lustre: MGC192.168.203.141@tcp: Connection restored to  (at 0@lo)
[ 5696.454748] Lustre: Skipped 19 previous similar messages
[ 5696.773626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
[ 5696.776730] Lustre: Skipped 7 previous similar messages
[ 5696.882212] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7235 to 0x0:7297
[ 5700.178970] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5706.210785] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 1 client reconnects
[ 5706.214451] Lustre: Skipped 7 previous similar messages
[ 5706.282664] Lustre: lustre-OST0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted.
[ 5706.290181] Lustre: Skipped 7 previous similar messages
[ 5706.299098] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6179 to 0x0:6273
[ 5708.437890] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5716.101912] Lustre: DEBUG MARKER: == replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0 ========================================================== 10:46:22 (1761317182)
[ 5717.119272] Lustre: DEBUG MARKER: SKIP: replay-single test_80a needs >= 2 MDTs
[ 5718.255298] Lustre: DEBUG MARKER: == replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 ========================================================== 10:46:24 (1761317184)
[ 5719.186405] Lustre: DEBUG MARKER: SKIP: replay-single test_80b needs >= 2 MDTs
[ 5720.433357] Lustre: DEBUG MARKER: == replay-single test 80c: DNE: create remote dir, drop update rep from MDT1, fail MDT[0,1] ========================================================== 10:46:26 (1761317186)
[ 5721.517608] Lustre: DEBUG MARKER: SKIP: replay-single test_80c needs >= 2 MDTs
[ 5722.956618] Lustre: DEBUG MARKER: == replay-single test 80d: DNE: create remote dir, drop update rep from MDT1, fail 2 MDTs ========================================================== 10:46:28 (1761317188)
[ 5724.080073] Lustre: DEBUG MARKER: SKIP: replay-single test_80d needs >= 2 MDTs
[ 5725.508555] Lustre: DEBUG MARKER: == replay-single test 80e: DNE: create remote dir, drop MDT1 rep, fail MDT0 ========================================================== 10:46:31 (1761317191)
[ 5726.647311] Lustre: DEBUG MARKER: SKIP: replay-single test_80e needs >= 2 MDTs
[ 5728.163767] Lustre: DEBUG MARKER: == replay-single test 80f: DNE: create remote dir, drop MDT1 rep, fail MDT1 ========================================================== 10:46:34 (1761317194)
[ 5729.872308] Lustre: DEBUG MARKER: SKIP: replay-single test_80f needs >= 2 MDTs
[ 5731.589749] Lustre: DEBUG MARKER: == replay-single test 80g: DNE: create remote dir, drop MDT1 rep, fail MDT0, then MDT1 ========================================================== 10:46:37 (1761317197)
[ 5732.810933] Lustre: DEBUG MARKER: SKIP: replay-single test_80g needs >= 2 MDTs
[ 5734.196568] Lustre: DEBUG MARKER: == replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs ========================================================== 10:46:40 (1761317200)
[ 5735.342544] Lustre: DEBUG MARKER: SKIP: replay-single test_80h needs >= 2 MDTs
[ 5736.669659] Lustre: DEBUG MARKER: == replay-single test 81a: DNE: unlink remote dir, drop MDT0 update rep, fail MDT1 ========================================================== 10:46:42 (1761317202)
[ 5737.732219] Lustre: DEBUG MARKER: SKIP: replay-single test_81a needs >= 2 MDTs
[ 5738.948745] Lustre: DEBUG MARKER: == replay-single test 81b: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0 ========================================================== 10:46:45 (1761317205)
[ 5740.275408] Lustre: DEBUG MARKER: SKIP: replay-single test_81b needs >= 2 MDTs
[ 5741.842519] Lustre: DEBUG MARKER: == replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 ========================================================== 10:46:47 (1761317207)
[ 5743.125320] Lustre: DEBUG MARKER: SKIP: replay-single test_81c needs >= 2 MDTs
[ 5744.398345] Lustre: DEBUG MARKER: == replay-single test 81d: DNE: unlink remote dir, drop MDT0 update reply, fail 2 MDTs ========================================================== 10:46:50 (1761317210)
[ 5745.555265] Lustre: DEBUG MARKER: SKIP: replay-single test_81d needs >= 2 MDTs
[ 5747.166358] Lustre: DEBUG MARKER: == replay-single test 81e: DNE: unlink remote dir, drop MDT1 req reply, fail MDT0 ========================================================== 10:46:52 (1761317212)
[ 5748.746527] Lustre: DEBUG MARKER: SKIP: replay-single test_81e needs >= 2 MDTs
[ 5750.629190] Lustre: DEBUG MARKER: == replay-single test 81f: DNE: unlink remote dir, drop MDT1 req reply, fail MDT1 ========================================================== 10:46:56 (1761317216)
[ 5752.375661] Lustre: DEBUG MARKER: SKIP: replay-single test_81f needs >= 2 MDTs
[ 5754.131193] Lustre: DEBUG MARKER: == replay-single test 81g: DNE: unlink remote dir, drop req reply, fail M0, then M1 ========================================================== 10:46:59 (1761317219)
[ 5755.562816] Lustre: DEBUG MARKER: SKIP: replay-single test_81g needs >= 2 MDTs
[ 5757.905329] Lustre: DEBUG MARKER: == replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs ========================================================== 10:47:03 (1761317223)
[ 5759.383950] Lustre: DEBUG MARKER: SKIP: replay-single test_81h needs >= 2 MDTs
[ 5761.270573] Lustre: DEBUG MARKER: == replay-single test 84a: stale open during export disconnect ========================================================== 10:47:06 (1761317226)
[ 5762.915740] Lustre: 168005:0:(genops.c:1710:obd_export_evict_by_uuid()) lustre-MDT0000: evicting 8ec3d07a-a9d0-47db-93fd-f6b301c0153c at adminstrative request
[ 5769.276926] Lustre: DEBUG MARKER: == replay-single test 85a: check the cancellation of unused locks during recovery(IBITS) ========================================================== 10:47:15 (1761317235)
[ 5793.159387] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6325 to 0x0:6369
[ 5793.165419] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7349 to 0x0:7393
[ 5794.585829] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5801.056710] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
[ 5802.239302] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
[ 5808.439398] Lustre: DEBUG MARKER: == replay-single test 85b: check the cancellation of unused locks during recovery(EXTENT) ========================================================== 10:47:54 (1761317274)
[ 5822.434197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping)
[ 5823.020370] Lustre: lustre-OST0000: Not available for connect from 192.168.203.41@tcp (stopping)
[ 5827.553540] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 5827.565850] LustreError: Skipped 4 previous similar messages
[ 5841.884444] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6470 to 0x0:6497
[ 5843.242447] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5850.558272] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5851.878834] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5860.343283] Lustre: DEBUG MARKER: == replay-single test 86: umount server after clear nid_stats should not hit LBUG ========================================================== 10:48:46 (1761317326)
[ 5871.610338] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7349 to 0x0:7425
[ 5871.619312] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6470 to 0x0:6529
[ 5877.845289] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5888.564262] Lustre: DEBUG MARKER: == replay-single test 87a: write replay ================== 10:49:13 (1761317353)
[ 5892.343322] LustreError: 172428:0:(osd_handler.c:698:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only ***
[ 5892.350581] LustreError: 172428:0:(osd_handler.c:698:osd_ro()) Skipped 7 previous similar messages
[ 5893.174735] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5896.673345] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107
[ 5914.152982] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6531 to 0x0:6561
[ 5915.584252] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5923.994983] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5925.311199] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5932.944893] Lustre: DEBUG MARKER: == replay-single test 87b: write replay with changed data (checksum resend) ========================================================== 10:49:58 (1761317398)
[ 5936.908220] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5958.508502] LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-192.168.203.41@tcp inode [0x200029441:0x5:0x0] object 0x0:6562 extent [0-1048575]: client csum f7275c8c, server csum 36527088
[ 5958.859611] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6563 to 0x0:6593
[ 5959.239276] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 5966.868242] Lustre: DEBUG MARKER: oleg341-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
[ 5968.025799] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
[ 5975.040210] Lustre: DEBUG MARKER: == replay-single test 88: MDS should not assign same objid to different files ========================================================== 10:50:40 (1761317440)
[ 5978.734908] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000
[ 5982.209917] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
[ 5991.640150] LustreError: 5500:0:(ldlm_lockd.c:2526:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1761317459 with bad export cookie 4976163742601084113
[ 6000.684404] LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 192.168.203.41@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
[ 6000.706677] LustreError: Skipped 19 previous similar messages
[ 6025.682940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect
[ 6025.701072] Lustre: Skipped 9 previous similar messages
[ 6029.247744] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6030.156936] Lustre: lustre-OST0001: deleting orphan objects from 0x0:7466 to 0x0:7489
[ 6047.346485] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6563 to 0x0:6625
[ 6048.105468] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6059.587558] Lustre: DEBUG MARKER: == replay-single test 89: no disk space leak on late ost connection ========================================================== 10:52:05 (1761317525)
[ 6108.640142] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6126.784287] Lustre: DEBUG MARKER: oleg341-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all 8
[ 6128.556802] Lustre: lustre-OST0000: Denying connection for new client a3bbe2a2-5d64-4ffb-97cc-83396511e7a4 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 1:06
[ 6128.567967] Lustre: Skipped 11 previous similar messages
[ 6138.909700] Lustre: lustre-OST0000: Denying connection for new client a3bbe2a2-5d64-4ffb-97cc-83396511e7a4 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:56
[ 6138.941063] Lustre: Skipped 1 previous similar message
[ 6159.389933] Lustre: lustre-OST0000: Denying connection for new client a3bbe2a2-5d64-4ffb-97cc-83396511e7a4 (at 192.168.203.41@tcp), waiting for 2 known clients (1 recovered, 0 in progress, and 0 evicted) to recover in 0:35
[ 6159.400502] Lustre: Skipped 3 previous similar messages
[ 6195.000722] Lustre: lustre-OST0000: recovery is timed out, evict stale exports
[ 6195.003368] Lustre: lustre-OST0000: disconnecting 1 stale clients
[ 6195.100600] Lustre: lustre-OST0000: deleting orphan objects from 0x0:6636 to 0x0:6657
[ 6197.595183] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 61 sec
[ 6210.391755] Lustre: DEBUG MARKER: free_before: 7518208 free_after: 7518208
[ 6215.437170] Lustre: DEBUG MARKER: == replay-single test 90: lfs find identifies the missing striped file segments ========================================================== 10:54:41 (1761317681)
[ 6308.999390] Lustre: DEBUG MARKER: replay-single test_90: @@@@@@ FAIL: wait_update OSTs up on MDT0000 failed