[ 1169.326652] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1169.447323] Lustre: Failing over lustre-MDT0000 [ 1172.028236] Lustre: server umount lustre-MDT0000 complete [ 1172.148905] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1172.148908] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.211430] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1172.217416] Lustre: lustre-MDT0000: Aborting client recovery [ 1172.219526] LustreError: 23573:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1172.226575] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.234025] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1172.243530] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1172.246028] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.657537] Lustre: 23703:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1172.665426] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557747/real 1717557747] req@ffff88009d55ca80 x1800988647554432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1173.403392] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557747/real 1717557747] req@ffff88009d55d180 x1800988647554368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557763 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1179.261283] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1186.185508] Lustre: DEBUG MARKER: == replay-single test 38: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:22:48 (1717557768) [ 1186.185513] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557760/real 1717557760] req@ffff8800a8161500 x1800988647559424/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1188.415340] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557760/real 1717557760] req@ffff8800a8160a80 x1800988647559488/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557776 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1188.720841] LustreError: 25502:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1189.441654] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1189.555723] Lustre: Failing over lustre-MDT0000 [ 1201.820683] Lustre: server umount lustre-MDT0000 complete [ 1201.928072] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1201.932767] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.857380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1204.727599] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1205.564408] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1206.093647] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1207.953652] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1207.994563] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff88009af84700 x1800988647584256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1212.961572] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557782/real 1717557782] req@ffff88009af85c00 x1800988647584320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1212.961638] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff88009af87480 x1800988647584512/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1214.605611] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557787/real 1717557787] req@ffff88009af85500 x1800988647584448/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1220.525318] Lustre: DEBUG MARKER: == replay-single test 39: test recovery from unlink llog (test llog_gen_rec) ========================================================== 23:23:23 (1717557803) [ 1220.866047] LustreError: 28615:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1224.436037] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1224.464201] Lustre: Failing over lustre-MDT0000 [ 1224.577610] LustreError: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8800a740a300 x1800988647643456/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'osp-syn-0-0.0' uid:0 gid:0 [ 1236.623225] Lustre: server umount lustre-MDT0000 complete [ 1236.745268] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1236.745585] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1237.858562] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1239.853596] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1241.697749] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1242.293965] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1242.753558] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1242.753566] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557817/real 1717557817] req@ffff8800a13bdf80 x1800988647645760/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1247.777564] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557817/real 1717557817] req@ffff8800a13be300 x1800988647645696/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1247.790500] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557822/real 1717557822] req@ffff8800a13bf100 x1800988647645888/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1251.077490] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557822/real 1717557822] req@ffff8800a13bd180 x1800988647645952/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557838 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1251.703741] Lustre: DEBUG MARKER: == replay-single test 41: read from a valid osc while other oscs are invalid ========================================================== 23:24:00 (1717557840) [ 1252.086284] Lustre: setting import lustre-OST0001_UUID INACTIVE by administrator request [ 1252.091186] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1252.098960] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1254.133764] LustreError: 167-0: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 1260.267135] Lustre: DEBUG MARKER: == replay-single test 42: recovery after ost failure ===== 23:24:03 (1717557843) [ 1260.606764] LustreError: 32632:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 1264.369738] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 1264.394056] Lustre: Failing over lustre-OST0000 [ 1265.257613] Lustre: server umount lustre-OST0000 complete [ 1265.818080] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.826850] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1270.260212] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1270.826087] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.268135] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1275.833876] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.290953] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1278.467843] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1324.044477] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1325.176230] Lustre: DEBUG MARKER: == replay-single test 43: mds osc import failure during recovery; don't LBUG ========================================================== 23:25:13 (1717557913) [ 1325.499949] LustreError: 3340:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1326.564662] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1326.671845] Lustre: Failing over lustre-MDT0000 [ 1338.518685] Lustre: server umount lustre-MDT0000 complete [ 1338.622445] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1338.622470] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1339.760604] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1340.384539] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1342.653516] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1342.665513] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1342.665522] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557917/real 1717557917] req@ffff8801353faa00 x1800988647762048/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1343.139231] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557917/real 1717557917] req@ffff8801353fb800 x1800988647761984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557933 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1347.657617] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1347.665535] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557922/real 1717557922] req@ffff8801353f8000 x1800988647762176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1352.665564] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557922/real 1717557922] req@ffff8801353fa300 x1800988647762240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1352.665569] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557927/real 1717557927] req@ffff88009cd0d180 x1800988647762432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1356.432602] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557927/real 1717557927] req@ffff8801353fad80 x1800988647762368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717557943 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1356.448178] Lustre: 4376:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717557930/real 1717557930] req@ffff88009d6d9180 x1800988647777664/t0(0) o5->lustre-OST0001-osc-MDT0000@0@lo:28/4 lens 432/432 e 0 to 1 dl 1717557946 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'osp-pre [ 1356.463431] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1356.464109] LustreError: 4376:0:(/home/green/git/lustre-release/lustre/osp/osp_precreate.c:992:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -11 [ 1356.650943] Lustre: lustre-OST0001: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1358.475048] Lustre: DEBUG MARKER: == replay-single test 44a: race in target handle connect ========================================================== 23:25:45 (1717557945) [ 1363.481601] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1363.488626] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1364.215555] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1369.218627] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1369.225344] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1369.963141] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1374.968599] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1374.978722] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1375.689108] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1380.693620] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1380.700763] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1381.426750] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1386.432676] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1386.440838] LustreError: 4374:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1387.132032] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1392.136585] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1392.140339] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1392.812253] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1397.818654] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1397.825151] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1398.521493] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1403.526695] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1403.532866] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1404.232001] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1409.238679] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1409.245798] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1409.956583] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1414.961551] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:197:cfs_race_loc()) cfs_race id 701 sleeping [ 1414.968217] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/include/libcfs/libcfs_fail.h:208:cfs_race_loc()) cfs_fail_race id 701 awake: rc=0 [ 1417.980504] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1418.549920] Lustre: DEBUG MARKER: == replay-single test 44b: race in target handle connect ========================================================== 23:26:47 (1717558007) [ 1418.556919] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1428.550562] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1433.556505] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1438.564107] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1443.572100] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1448.580103] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1453.588254] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1458.561591] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1458.566600] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1458.596142] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88012c3a4380 x1800988642732224/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558029 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1459.314762] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1459.319639] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1469.315508] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1474.324401] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1479.332058] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1480.024858] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1484.340327] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1489.348406] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1494.356368] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1499.323580] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1499.328535] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1499.364017] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8800953bb800 x1800988642734272/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558069 ref 1 fl Complete:H/200/0 rc 0/0 job:'lc [ 1499.380393] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1524.364510] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1529.380060] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1534.388136] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1539.385542] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1539.390512] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1539.398200] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff8800a8162680 x1800988642735424/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558109 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1539.413218] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1564.396830] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1569.412329] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1574.420193] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1579.417599] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1579.423749] LustreError: 4331:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1579.427856] Lustre: 4331:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/20s); client may timeout req@ffff88009ce2fb80 x1800988642736320/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558149 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1579.427866] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1604.428189] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1609.444093] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1614.452349] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.460547] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.527562] Lustre: lustre-MDT0000: Export ffff8800947c8000 already connecting from 192.168.203.45@tcp [ 1619.534558] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 awake [ 1624.467732] Lustre: 4332:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/21s); client may timeout req@ffff88009b3d1500 x1800988642737216/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558189 ref 1 fl Complete:H/200/0 rc 0/0 job:'kw [ 1624.479601] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1645.282517] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 704 sleeping for 40000ms [ 1645.291742] LustreError: 4332:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:148:__cfs_fail_timeout_set()) cfs_fail_timeout interrupted [ 1646.809590] Lustre: 4332:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:2359:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (20/1s); client may timeout req@ffff88012b0f2680 x1800988642738304/t0(0) o38->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:0/0 lens 520/416 e 0 to 0 dl 1717558234 ref 1 fl Complete:H/200/0 rc 0/0 job:'kwo [ 1647.790645] Lustre: DEBUG MARKER: == replay-single test 44c: race in target handle connect ========================================================== 23:30:36 (1717558236) [ 1648.132227] LustreError: 9524:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1649.623640] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1649.768696] Lustre: Failing over lustre-MDT0000 [ 1652.208486] Lustre: server umount lustre-MDT0000 complete [ 1652.300719] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.342805] LustreError: 19025:0:(/home/green/git/lustre-release/lustre/ptlrpc/service.c:1236:ptlrpc_check_req()) @@@ Invalid replay without recovery req@ffff880131d58a80 x1800988647810560/t0(0) o400->lustre-MDT0000-mdtlov_UUID@0@lo:0/0 lens 224/0 e 0 to 0 dl 0 ref 1 fl New:/2c0/ffffffff rc 0/-1 job:'ptlrpcd_rcv.0' uid:0 gid:0 [ 1652.353282] Lustre: lustre-MDT0000: Aborting client recovery [ 1652.355388] LustreError: 10624:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2927:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery [ 1652.359552] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1652.363576] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/obdclass/genops.c:1533:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client a372da69-b38f-485d-a31f-21d31dc1fbfe@ [ 1652.368632] Lustre: lustre-MDT0000: disconnecting 1 stale clients [ 1652.370211] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1653.472249] Lustre: 10763:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:2310:target_recovery_overseer()) recovery is aborted, evict exports in recovery [ 1657.322007] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1657.322168] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.553742] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1657.681845] Lustre: Failing over lustre-MDT0000 [ 1670.147271] Lustre: server umount lustre-MDT0000 complete [ 1670.312562] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1670.312580] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1671.480623] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1673.546345] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1674.414844] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1675.003957] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1678.307845] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1678.337541] Lustre: DEBUG MARKER: == replay-single test 45: Handle failed close ============ 23:31:07 (1717558267) [ 1678.337552] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558252/real 1717558252] req@ffff8800a0765180 x1800988647814912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1678.361428] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558252/real 1717558252] req@ffff8800a0764700 x1800988647814976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1678.440151] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1681.609876] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1681.963025] Lustre: DEBUG MARKER: == replay-single test 46: Don't leak file handle after open resend (3325) ========================================================== 23:31:10 (1717558270) [ 1683.329598] LustreError: 12193:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a6c65180 x1800988642763392/t0(0) o700->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:478/0 lens 264/248 e 0 to 0 dl 1717558283 ref 1 fl Interpret:/200/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 1683.345695] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558257/real 1717558257] req@ffff8800a0764000 x1800988647815104/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1697.985787] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558257/real 1717558257] req@ffff8800a0765880 x1800988647815168/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1699.078769] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 1699.219007] Lustre: Failing over lustre-MDT0000 [ 1711.642259] Lustre: server umount lustre-MDT0000 complete [ 1711.792092] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1711.792121] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.793575] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.799560] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558242/real 1717558242] req@ffff8800a0766a00 x1800988647809920/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1712.989997] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558242/real 1717558242] req@ffff8800a0766d80 x1800988647809984/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558303 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1714.045048] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1715.791607] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1715.840545] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558290/real 1717558290] req@ffff8800a7375180 x1800988647821888/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1715.909482] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558290/real 1717558290] req@ffff8800a7377800 x1800988647821824/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558306 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1716.513360] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1719.297819] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1720.188774] Lustre: DEBUG MARKER: == replay-single test 47: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:31:48 (1717558308) [ 1720.211093] Lustre: Failing over lustre-OST0000 [ 1721.188629] Lustre: server umount lustre-OST0000 complete [ 1721.817975] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558295/real 1717558295] req@ffff8800a7376680 x1800988647822016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558311 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1721.835865] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1721.840813] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.052302] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1725.833571] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.841842] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558300/real 1717558300] req@ffff8800a6c65880 x1800988647822208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558316 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1729.044007] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1731.849768] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.052192] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1734.314198] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 1736.968994] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1737.529642] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1802.646940] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 1803.817396] Lustre: DEBUG MARKER: == replay-single test 48: MDS->OSC failure during precreate cleanup (2824) ========================================================== 23:33:11 (1717558391) [ 1804.143883] LustreError: 17865:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1804.980222] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1805.112040] Lustre: Failing over lustre-MDT0000 [ 1817.601012] Lustre: server umount lustre-MDT0000 complete [ 1817.705870] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1817.705917] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1818.932270] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.189587] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1823.793600] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1823.793618] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558398/real 1717558398] req@ffff88012d4ab100 x1800988647838784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1828.801629] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558398/real 1717558398] req@ffff880094aef480 x1800988647838720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558414 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1828.801640] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558403/real 1717558403] req@ffff88009b3d3480 x1800988647838976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1883.870872] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558403/real 1717558403] req@ffff88009b3d1c00 x1800988647838912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558419 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1884.594549] Lustre: DEBUG MARKER: == replay-single test 50: Double OSC recovery, don't LASSERT (3812) ========================================================== 23:34:33 (1717558473) [ 1884.604082] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1884.953864] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1884.959132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1892.151836] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 1893.205035] Lustre: DEBUG MARKER: == replay-single test 52: time out lock replay (3764) ==== 23:34:41 (1717558481) [ 1893.376789] Lustre: Failing over lustre-MDT0000 [ 1905.247472] Lustre: server umount lustre-MDT0000 complete [ 1905.367697] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1905.367763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1906.565380] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1908.664311] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1908.672974] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1914.381497] LustreError: 21733:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009de84a80 x1800988642818240/t0(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:705/0 lens 328/344 e 0 to 0 dl 1717558510 ref 1 fl Complete:/240/0 rc 0/0 job:'ldlm_lock_repla.0' uid:0 gid:0 [ 1914.387690] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558488/real 1717558488] req@ffff8801353fad80 x1800988647854400/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1919.401572] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558488/real 1717558488] req@ffff8801353f8000 x1800988647854336/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558504 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1919.401611] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558493/real 1717558493] req@ffff8801353f9c00 x1800988647854528/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1924.355978] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558493/real 1717558493] req@ffff8801353faa00 x1800988647854592/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558509 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1924.860207] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:54 [ 1925.205227] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1929.081027] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1931.863533] Lustre: DEBUG MARKER: == replay-single test 53a: |X| close request while two MDC requests in flight ========================================================== 23:35:18 (1717558518) [ 1932.190605] LustreError: 23165:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1932.912245] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1933.058273] Lustre: Failing over lustre-MDT0000 [ 1945.229126] Lustre: server umount lustre-MDT0000 complete [ 1945.359624] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1945.359648] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.498022] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1946.500654] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1949.447052] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1949.973088] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1951.361555] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1951.377511] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558525/real 1717558525] req@ffff8800a8161f80 x1800988647861312/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1953.607806] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558525/real 1717558525] req@ffff8800a8162a00 x1800988647861376/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558541 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1955.984972] Lustre: DEBUG MARKER: == replay-single test 53b: |X| open request while two MDC requests in flight ========================================================== 23:35:42 (1717558542) [ 1956.247032] LustreError: 25576:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1956.364530] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1956.385531] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558530/real 1717558530] req@ffff8800a8162680 x1800988647861568/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1956.926261] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558530/real 1717558530] req@ffff8800a8161180 x1800988647861504/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558546 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1957.093407] Lustre: Failing over lustre-MDT0000 [ 1969.660498] Lustre: server umount lustre-MDT0000 complete [ 1969.808565] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1969.808780] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.068588] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1970.931669] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 1973.817091] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 1974.396175] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1975.825512] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 1975.835493] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558550/real 1717558550] req@ffff88012c3a6a00 x1800988647867136/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1977.670978] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558550/real 1717558550] req@ffff88012c3a5f80 x1800988647867200/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558566 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1980.444689] Lustre: DEBUG MARKER: == replay-single test 53c: |X| open request and close request while two MDC requests in flight ========================================================== 23:36:06 (1717558566) [ 1980.731247] LustreError: 27851:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 1980.833553] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 1981.100524] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558555/real 1717558555] req@ffff88012c3a5180 x1800988647867392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1981.414622] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558555/real 1717558555] req@ffff88012c3a7480 x1800988647867328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558571 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 1981.576179] Lustre: Failing over lustre-MDT0000 [ 1994.089774] Lustre: server umount lustre-MDT0000 complete [ 1994.127210] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.238793] LustreError: 137-5: lustre-MDT0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1994.247349] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.182951] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1995.455842] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2000.498075] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2001.273638] Lustre: DEBUG MARKER: == replay-single test 53d: close reply while two MDC requests in flight ========================================================== 23:36:29 (1717558589) [ 2001.273646] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558575/real 1717558575] req@ffff8800953ff800 x1800988647872960/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2001.889617] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558575/real 1717558575] req@ffff8800953fea00 x1800988647873024/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558591 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2002.916410] LustreError: 28712:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009b8c2a00 x1800988642839488/t257698037777(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:43/0 lens 392/456 e 0 to 0 dl 1717558603 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2003.049571] Lustre: Failing over lustre-MDT0000 [ 2005.916676] Lustre: server umount lustre-MDT0000 complete [ 2006.281665] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558580/real 1717558580] req@ffff8800953fdf80 x1800988647873152/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2015.261678] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558580/real 1717558580] req@ffff880094aef480 x1800988647873216/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558596 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2015.420966] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2015.424186] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2016.476673] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2017.920894] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2017.954663] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2019.236309] Lustre: 30703:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a12d4000 x1800988642839488/t257698037777(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:59/0 lens 392/456 e 0 to 0 dl 1717558619 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2019.736376] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2020.441573] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2021.457577] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558594/real 1717558594] req@ffff8800a13bd500 x1800988647878336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2023.474309] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558594/real 1717558594] req@ffff8800a13bdf80 x1800988647878272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2023.876205] Lustre: DEBUG MARKER: == replay-single test 53e: |X| open reply while two MDC requests in flight ========================================================== 23:36:52 (1717558612) [ 2025.457607] LustreError: 30701:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a13be300 x1800988642845504/t261993005072(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:65/0 lens 504/448 e 0 to 0 dl 1717558625 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2026.054895] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558599/real 1717558599] req@ffff8800a13bf100 x1800988647878464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2026.402188] LustreError: 32187:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2026.465550] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2027.153247] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558599/real 1717558599] req@ffff8800a13bc000 x1800988647878528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2027.293604] Lustre: Failing over lustre-MDT0000 [ 2030.152577] Lustre: server umount lustre-MDT0000 complete [ 2030.152637] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558604/real 1717558604] req@ffff88009ce2fb80 x1800988647878720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2039.695847] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558604/real 1717558604] req@ffff8800a13bed80 x1800988647878656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558620 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2039.846907] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2039.847001] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2040.996990] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.064812] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2041.099092] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2043.623037] Lustre: 593:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880131d59500 x1800988642845504/t261993005072(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:82/0 lens 504/2880 e 0 to 0 dl 1717558642 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2044.228287] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2045.862599] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2045.873523] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558620/real 1717558620] req@ffff88012c3a4e00 x1800988647884608/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2047.850013] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558620/real 1717558620] req@ffff88012c3a5f80 x1800988647884544/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558636 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2048.261870] Lustre: DEBUG MARKER: == replay-single test 53f: |X| open reply and close reply while two MDC requests in flight ========================================================== 23:37:17 (1717558637) [ 2049.602797] LustreError: 594:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7374380 x1800988642851584/t266287972368(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:89/0 lens 504/448 e 0 to 0 dl 1717558649 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2050.657275] LustreError: 597:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7375880 x1800988642851712/t266287972369(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:91/0 lens 392/456 e 0 to 0 dl 1717558651 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2050.881554] LustreError: 2085:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2050.881570] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558625/real 1717558625] req@ffff88009ce2df80 x1800988647884800/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2051.024874] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558625/real 1717558625] req@ffff88009ce2fb80 x1800988647884736/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558641 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2051.687659] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2051.840592] Lustre: Failing over lustre-MDT0000 [ 2063.943151] Lustre: server umount lustre-MDT0000 complete [ 2064.057118] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2064.061613] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.288769] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2064.335308] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2064.335341] Lustre: 2988:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a7375180 x1800988642851584/t266287972368(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:105/0 lens 504/2880 e 0 to 0 dl 1717558665 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2065.259341] Lustre: 2946:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a7374700 x1800988642851712/t266287972369(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:105/0 lens 392/456 e 0 to 0 dl 1717558665 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2070.788261] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2071.081574] Lustre: DEBUG MARKER: == replay-single test 53g: |X| drop open reply and close request while close and open are both in flight ========================================================== 23:37:39 (1717558659) [ 2071.196389] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558645/real 1717558645] req@ffff88012f1b2d80 x1800988647890176/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2071.327629] LustreError: 2943:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff8800a7377480 x1800988642857280/t270582939664(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:112/0 lens 504/448 e 0 to 0 dl 1717558672 ref 1 fl Interpret:/200/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2073.868667] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558645/real 1717558645] req@ffff88012f1b1180 x1800988647890240/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558661 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2074.181234] LustreError: 4429:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2074.867622] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2075.015218] Lustre: Failing over lustre-MDT0000 [ 2075.867588] Lustre: server umount lustre-MDT0000 complete [ 2075.867652] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558650/real 1717558650] req@ffff88012f1b2a00 x1800988647890368/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2087.414852] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558650/real 1717558650] req@ffff880094389c00 x1800988647890432/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558666 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2087.531996] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2087.543698] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2088.697360] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2094.532618] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2094.569598] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558669/real 1717558669] req@ffff88012f1b3b80 x1800988647895744/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2096.254025] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558669/real 1717558669] req@ffff88012f1b1f80 x1800988647895808/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2096.297715] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2099.577643] Lustre: 5337:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009de87100 x1800988642857280/t270582939664(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:137/0 lens 504/2880 e 0 to 0 dl 1717558697 ref 1 fl Interpret:/202/0 rc 0/0 job:'mcreate.0' uid:0 gid:0 [ 2099.577651] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558674/real 1717558674] req@ffff8800a6986300 x1800988647896000/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2100.664277] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558674/real 1717558674] req@ffff8800a6985500 x1800988647895936/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558690 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2102.384328] Lustre: DEBUG MARKER: == replay-single test 53h: open request and close reply while two MDC requests in flight ========================================================== 23:38:09 (1717558689) [ 2104.350361] LustreError: 5340:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009d55c000 x1800988642863232/t274877906960(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:143/0 lens 392/456 e 0 to 0 dl 1717558703 ref 1 fl Interpret:/200/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2104.689137] LustreError: 6748:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2105.347729] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2105.476673] Lustre: Failing over lustre-MDT0000 [ 2117.773695] Lustre: server umount lustre-MDT0000 complete [ 2118.925823] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2122.938280] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2122.946275] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.922840] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2123.930320] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558698/real 1717558698] req@ffff88009d55ea00 x1800988647902016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2127.126030] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558698/real 1717558698] req@ffff88009d55ed80 x1800988647902080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558714 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2127.167633] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2128.945558] Lustre: 7643:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009de86300 x1800988642863232/t274877906960(0) o35->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:168/0 lens 392/456 e 0 to 0 dl 1717558728 ref 1 fl Interpret:/202/0 rc 0/0 job:'multiop.0' uid:0 gid:0 [ 2128.945567] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558703/real 1717558703] req@ffff88009d55f480 x1800988647902272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2131.748329] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558703/real 1717558703] req@ffff88009d55c000 x1800988647902208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558719 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2132.059535] Lustre: DEBUG MARKER: == replay-single test 55: let MDS_CHECK_RESENT return the original return code instead of 0 ========================================================== 23:38:41 (1717558721) [ 2133.953606] LustreError: 8204:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88009de86a00 x1800988642867584/t279172874255(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:173/0 lens 664/608 e 0 to 0 dl 1717558733 ref 1 fl Interpret:/200/0 rc 301/0 job:'touch.0' uid:0 gid:0 [ 2133.953612] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558708/real 1717558708] req@ffff88009d55f100 x1800988647902464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2148.082207] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558708/real 1717558708] req@ffff88009d55f800 x1800988647902400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558724 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2148.100082] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2150.328414] Lustre: 8204:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff8800a6ced180 x1800988642867584/t279172874255(0) o101->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:189/0 lens 664/3488 e 0 to 0 dl 1717558749 ref 1 fl Interpret:/202/0 rc 0/0 job:'touch.0' uid:0 gid:0 [ 2151.403611] Lustre: DEBUG MARKER: == replay-single test 56: don't replay a symlink open request (3440) ========================================================== 23:38:59 (1717558739) [ 2151.698070] LustreError: 9416:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2152.404040] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2152.534199] Lustre: Failing over lustre-MDT0000 [ 2165.016710] Lustre: server umount lustre-MDT0000 complete [ 2165.170262] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2165.170301] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2166.389841] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.118179] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2169.190533] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2169.196265] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558743/real 1717558743] req@ffff8800a7723800 x1800988647910080/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2169.285429] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558743/real 1717558743] req@ffff8800a7720380 x1800988647910016/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558759 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2169.814069] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2174.179591] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2174.185525] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558748/real 1717558748] req@ffff8800a7722300 x1800988647910208/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2179.209550] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558748/real 1717558748] req@ffff8800a7720e00 x1800988647910272/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558764 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2179.209577] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558753/real 1717558753] req@ffff8800a7722680 x1800988647910400/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2182.420272] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558753/real 1717558753] req@ffff8800a7722d80 x1800988647910464/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558769 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2183.616003] Lustre: DEBUG MARKER: == replay-single test 57: test recovery from llog for setattr op ========================================================== 23:39:31 (1717558771) [ 2183.834243] LustreError: 11787:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2184.304802] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2184.393919] Lustre: Failing over lustre-MDT0000 [ 2195.919799] Lustre: server umount lustre-MDT0000 complete [ 2196.018266] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2196.018268] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2196.730582] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2198.179713] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2198.911421] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2199.251980] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2200.916867] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2201.029491] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 1475 [ 2201.043524] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558775/real 1717558775] req@ffff880093e19c00 x1800988647916032/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2205.423256] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558775/real 1717558775] req@ffff880093e1bb80 x1800988647916096/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558791 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2206.023510] Lustre: DEBUG MARKER: == replay-single test 58a: test recovery from llog for setattr op (test llog_gen_rec) ========================================================== 23:39:54 (1717558794) [ 2206.033582] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558780/real 1717558780] req@ffff88009a274700 x1800988647916224/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2211.041493] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558780/real 1717558780] req@ffff88009a276680 x1800988647916288/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558796 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2211.041499] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558785/real 1717558785] req@ffff88009a277480 x1800988647916416/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2212.496008] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558785/real 1717558785] req@ffff88009a274e00 x1800988647916480/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558801 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2212.733310] LustreError: 14867:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2213.204712] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2213.341297] Lustre: Failing over lustre-MDT0000 [ 2224.895792] Lustre: server umount lustre-MDT0000 complete [ 2224.997436] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2224.997857] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2225.747157] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.436472] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2227.996655] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2228.337574] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2231.999478] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2232.025485] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558806/real 1717558806] req@ffff8800a0863b80 x1800988647927360/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2237.017512] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558806/real 1717558806] req@ffff8800a0860a80 x1800988647927424/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558822 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2237.033496] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558811/real 1717558811] req@ffff8800a0862680 x1800988647927552/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2241.449346] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558811/real 1717558811] req@ffff8800a0860700 x1800988647927616/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558827 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2242.312486] Lustre: DEBUG MARKER: == replay-single test 58b: test replay of setxattr op ==== 23:40:30 (1717558830) [ 2242.671626] LustreError: 17704:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2243.165124] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2243.282528] Lustre: Failing over lustre-MDT0000 [ 2254.799945] Lustre: server umount lustre-MDT0000 complete [ 2254.924874] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2254.924949] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2255.708032] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.564647] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2257.901323] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2258.225407] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2260.266676] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2260.611835] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount FULL mgc.*.mgs_server_uuid [ 2260.945487] Lustre: DEBUG MARKER: mgc.*.mgs_server_uuid in FULL state after 0 sec [ 2260.945492] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558835/real 1717558835] req@ffff8800a0942680 x1800988648187392/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2261.962036] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558835/real 1717558835] req@ffff8800a0943800 x1800988648187328/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558851 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2265.953549] Lustre: DEBUG MARKER: == replay-single test 58c: resend/reconstruct setxattr op ========================================================== 23:40:51 (1717558851) [ 2265.953556] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558840/real 1717558840] req@ffff88009a8ff800 x1800988648187584/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2283.372384] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558840/real 1717558840] req@ffff88009a8fce00 x1800988648187520/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558856 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2283.885380] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2299.893473] LustreError: 18764:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/ldlm/ldlm_lib.c:3271:target_send_reply_msg()) @@@ dropping reply req@ffff88008f240a80 x1800988644335360/t296352743433(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:325/0 lens 66040/440 e 0 to 0 dl 1717558885 ref 1 fl Interpret:/200/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2299.907269] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnecting [ 2301.833357] Lustre: 20085:0:(/home/green/git/lustre-release/lustre/mdt/mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff88009b8c1180 x1800988644335360/t296352743433(0) o36->a372da69-b38f-485d-a31f-21d31dc1fbfe@192.168.203.45@tcp:341/0 lens 66040/440 e 0 to 0 dl 1717558901 ref 1 fl Interpret:/202/0 rc 0/0 job:'setfattr.0' uid:0 gid:0 [ 2302.195106] Lustre: DEBUG MARKER: SKIP: replay-single test_59 skipping ALWAYS excluded test 59 [ 2304.054960] Lustre: DEBUG MARKER: == replay-single test 60: test llog post recovery init vs llog unlink ========================================================== 23:41:31 (1717558891) [ 2304.273225] LustreError: 21395:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2304.875795] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2304.964123] Lustre: Failing over lustre-MDT0000 [ 2317.079934] Lustre: server umount lustre-MDT0000 complete [ 2317.228872] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2317.229125] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2318.397267] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2321.237642] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2321.282633] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558895/real 1717558895] req@ffff88009b0eaa00 x1800988648390272/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2325.932531] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558895/real 1717558895] req@ffff88009b0e9c00 x1800988648390336/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558911 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2326.244551] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2327.085051] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558900/real 1717558900] req@ffff88009b0ebb80 x1800988648390528/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2327.256592] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2327.669394] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558900/real 1717558900] req@ffff88009b0eb480 x1800988648390464/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2331.590118] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2332.239602] Lustre: DEBUG MARKER: == replay-single test 61a: test race llog recovery vs llog cleanup ========================================================== 23:42:00 (1717558920) [ 2332.273597] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558905/real 1717558905] req@ffff88009b0ea680 x1800988648390656/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2336.816752] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717558905/real 1717558905] req@ffff88009b0ead80 x1800988648390720/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717558921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2337.028007] LustreError: 24046:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-OST0000: *** setting device osd-zfs read-only *** [ 2340.910698] Lustre: DEBUG MARKER: ost1 REPLAY BARRIER on lustre-OST0000 [ 2340.932186] Lustre: Failing over lustre-OST0000 [ 2341.246926] Lustre: server umount lustre-OST0000 complete [ 2341.247310] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247348] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247430] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2341.247473] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247631] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.247818] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.249703] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2341.287744] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2341.291136] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2342.297677] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 2344.970286] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2347.306092] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2349.972149] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2352.313875] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2354.442802] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2355.032331] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2366.455779] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2366.475922] Lustre: Failing over lustre-OST0000 [ 2368.298198] Lustre: server umount lustre-OST0000 complete [ 2368.301216] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2369.021934] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2373.305958] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.020200] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2378.313806] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2379.027694] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2380.617282] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2383.496251] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2384.087103] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2417.897689] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2418.918644] Lustre: DEBUG MARKER: == replay-single test 61b: test race mds llog sync vs llog cleanup ========================================================== 23:43:27 (1717559007) [ 2419.067011] Lustre: Failing over lustre-MDT0000 [ 2431.539170] Lustre: server umount lustre-MDT0000 complete [ 2431.685661] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2431.689593] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2432.810794] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2434.140047] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2439.690687] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2439.713584] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559014/real 1717559014] req@ffff880131d5bb80 x1800988648474624/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2444.181952] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559014/real 1717559014] req@ffff880131d59180 x1800988648474688/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559030 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2444.323609] Lustre: Failing over lustre-MDT0000 [ 2444.721566] Lustre: server umount lustre-MDT0000 complete [ 2445.181587] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559019/real 1717559019] req@ffff880131d5b800 x1800988648474816/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2456.590069] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559019/real 1717559019] req@ffff880131d58000 x1800988648474880/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559035 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2456.746031] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.746062] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2457.917389] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.187398] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2460.866154] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2461.471196] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2462.769582] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2462.785589] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559037/real 1717559037] req@ffff880091835180 x1800988648480128/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2464.996930] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559037/real 1717559037] req@ffff880091837800 x1800988648480064/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559053 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2467.793604] Lustre: DEBUG MARKER: == replay-single test 61c: test race mds llog sync vs llog cleanup ========================================================== 23:44:14 (1717559054) [ 2467.801415] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559042/real 1717559042] req@ffff8800a0893100 x1800988648480256/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2476.257815] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559042/real 1717559042] req@ffff8800a0893800 x1800988648480320/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2476.287876] Lustre: Failing over lustre-OST0000 [ 2476.793915] Lustre: server umount lustre-OST0000 complete [ 2476.796595] LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2476.801474] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2479.210899] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2481.817860] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2484.196026] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2486.825828] LustreError: 137-5: lustre-OST0000: not available for connect from 192.168.203.45@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.203790] LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2490.320028] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2493.198605] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2493.791178] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2497.750390] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 2498.469878] Lustre: DEBUG MARKER: == replay-single test 61d: error in llog_setup should cleanup the llog context correctly ========================================================== 23:44:46 (1717559086) [ 2498.592899] Lustre: Failing over lustre-MDT0000 [ 2501.725831] Lustre: server umount lustre-MDT0000 complete [ 2501.729017] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/llog_obd.c:207:llog_setup()) MGS: ctxt 0 lop_setup=ffffffffa0451350 failed: rc = -95 [ 2501.736668] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/obd_config.c:797:class_setup()) setup MGS failed (-95) [ 2501.743341] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/obdclass/obd_mount.c:216:lustre_start_simple()) MGS setup error -95 [ 2501.749973] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) MGS not registered [ 2501.756839] LustreError: 0-0: Failed to start MGS 'MGS' (-95). Is the 'mgs' module loaded? [ 2501.761212] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:1751:server_put_super()) no obd lustre-MDT0000 [ 2501.779651] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/ptlrpc/../../lustre/target/tgt_mount.c:135:server_deregister_mount()) lustre-MDT0000 not registered [ 2501.785509] Lustre: server umount lustre-MDT0000 complete [ 2503.814882] LustreError: 2255:0:(/home/green/git/lustre-release/lustre/llite/super25.c:189:lustre_fill_super()) llite: Unable to mount : rc = -95 [ 2504.257198] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2505.138276] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2508.041971] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2508.986015] Lustre: DEBUG MARKER: == replay-single test 62: don't mis-drop resent replay === 23:44:57 (1717559097) [ 2508.986086] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.149978] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2509.486265] LustreError: 3728:0:(/home/green/git/lustre-release/lustre/osd-zfs/osd_handler.c:717:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** [ 2510.677249] Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 [ 2510.813403] Lustre: Failing over lustre-MDT0000 [ 2514.676626] Lustre: server umount lustre-MDT0000 complete [ 2514.679356] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559089/real 1717559089] req@ffff8801353f8a80 x1800988648489856/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2519.696596] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559089/real 1717559089] req@ffff8801353fad80 x1800988648489792/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559105 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2520.001592] Lustre: 3211:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559094/real 1717559094] req@ffff8801353fa300 x1800988648490048/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2523.371298] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559094/real 1717559094] req@ffff8801353f8000 x1800988648489984/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559110 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2523.545894] LustreError: 166-1: MGC192.168.203.145@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2523.554141] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.291164] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2524.755290] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects [ 2529.569540] Lustre: DEBUG MARKER: oleg345-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 2529.569570] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559104/real 1717559104] req@ffff8800a8163800 x1800988648494720/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2534.577580] Lustre: 3212:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559104/real 1717559104] req@ffff8800a8160a80 x1800988648494784/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559120 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2534.577647] Lustre: 3213:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559109/real 1717559109] req@ffff8800a8160e00 x1800988648494912/t0(0) o400->lustre-MDT0000-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2540.296170] Lustre: 3214:0:(/home/green/git/lustre-release/lustre/ptlrpc/client.c:2343:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1717559109/real 1717559109] req@ffff8800a8161f80 x1800988648494976/t0(0) o400->lustre-MDT0000-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1717559125 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kwor [ 2541.565919] Lustre: lustre-MDT0000: Client a372da69-b38f-485d-a31f-21d31dc1fbfe (at 192.168.203.45@tcp) reconnected, waiting for 1 clients in recovery for 0:53 [ 2542.138990] Lustre: DEBUG MARKER: oleg345-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2546.201972] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 2569.638258] Lustre: DEBUG MARKER: == replay-single test 65a: AT: verify early replies ====== 23:45:35 (1717559135) [ 2580.642632] LustreError: 5271:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:141:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a sleeping for 11000ms [ 2581.770997] LustreError: 5271:0:(/home/green/git/lustre-release/libcfs/libcfs/fail.c:154:__cfs_fail_timeout_set()) cfs_fail_timeout id 50a awake